date:20160527

[jira] [Created] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

2016-05-27 Thread Mohit Sabharwal (JIRA)

Mohit Sabharwal created HIVE-13884:
--

 Summary: Disallow queries fetching more than a configured number 
of partitions in PartitionPruner
 Key: HIVE-13884
 URL: https://issues.apache.org/jira/browse/HIVE-13884
 Project: Hive
  Issue Type: Improvement
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal


Currently the PartitionPruner requests either all partitions or partitions 
based on filter expression. In either scenarios, if the number of partitions 
accessed is large there can be significant memory pressure at the HMS server 
end.

We already have a config {{hive.limit.query.max.table.partition}} that enforces 
limits on number of partitions that may be scanned per operator. But this check 
happens after the PartitionPruner has already fetched all partitions.

We should add an option at PartitionPruner level to disallow queries that 
attempt to access number of partitions beyond a configurable limit.

Note that {{hive.mapred.mode=strict}} disallow queries without a partition 
filter in PartitionPruner, but this check accepts any query with a pruning 
condition, even if partitions fetched are large. In multi-tenant environments, 
admins could use more control w.r.t. number of partitions allowed based on HMS 
memory capacity.

One option is to have PartitionPruner first fetch the partition names (instead 
of partition specs) and throw an exception if number of partitions exceeds the 
configured value. Otherwise, fetch the partition specs.

Looks like the existing {{listPartitionNames}} call could be used if extended 
to take partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [discuss] jdk8 support

2016-05-27 Thread Sergio Pena

I did the change to JDK8 on Ptest now.

Please let me tknow if there are issues with Java8.

- Sergio

On Fri, May 27, 2016 at 2:45 PM, Sergio Pena 
wrote:

> Thanks Mohit.
>
> I will plan to do the JDK8 change on Jenkins today EOD and monitor any
> issues through the weekend.
>
> - Sergio
>
>
> On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal 
> wrote:
>
>> Update on moving Hive2 tests to JDK8: I've addressed almost all test
>> failures in HIVE-13547 on java8 branch. There is one remaining
>> open item (HIVE-13834) which is currently assigned. Given current
>> state of flaky test runs, there might be few more.
>>
>> I will work with Sergio to merge the test fixes to master and switch
>> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
>>
>> After Hive2 tests switch, if your patch sees ordering related test
>> failures in pre-commit run, it's likely JDK version related and you'll
>> need to build & re-run the test using JDK8. Number of such tests
>> should be relatively small.
>>
>> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal 
>> wrote:
>>
>> > Created HIVE-13547 to track switching 2x tests to JDK8.
>> >
>> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena > >
>> > wrote:
>> >
>> >> I agree with such change as JDK7 is not longer supported.
>> >>
>> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
>> >> replace the path from java7 to java8. But I think we should fix all
>> JDK8
>> >> issues or most of them before doing the change or we will end up
>> having a
>> >> lot of failures on all JIRAs running pre-commit tests.
>> >>
>> >> +1 with the change.
>> >>
>> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth 
>> wrote:
>> >>
>> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
>> >> requirement
>> >> > soon as well.
>> >> >
>> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho 
>> wrote:
>> >> >
>> >> > > Sounds like a good plan, +1
>> >> > >
>> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
>> mo...@cloudera.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target as
>> 1.7
>> >> > since
>> >> > > > we're not using any JDK8 specific features yet. So, option (3)
>> >> > > essentially
>> >> > > > just means using JDK8 exclusively to build & test Hive2.
>> >> > > >
>> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
>> >> > hashut...@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hi Mohit,
>> >> > > > >
>> >> > > > > I also think option 3 makes sense. We should strive to keep
>> test
>> >> > matrix
>> >> > > > > small so that we can do fast QA runs for dev patches.
>> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build &
>> >> test
>> >> > > > Hive2.
>> >> > > > > However, I am not sure of explicitly dropping support
>> altogether
>> >> for
>> >> > > jdk7
>> >> > > > > on Hive2. We should make sure that in pom.xml java source &
>> target
>> >> > > > > compatibility is still 1.7 (which already is the case
>> currently)
>> >> so
>> >> > > that
>> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we
>> can
>> >> run
>> >> > on
>> >> > > > > jdk8.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > Ashutosh
>> >> > > > >
>> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
>> >> mo...@cloudera.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Hi folks,
>> >> > > > > >
>> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April
>> 2015.
>> >> > > > > > In order to support Java 8, we fixed quite a bunch of tests
>> in
>> >> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
>> >> tests
>> >> > > run
>> >> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As
>> a
>> >> > > result,
>> >> > > > > > the count has crept up
>> >> > > > > > <
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
>> >> > > > > > >
>> >> > > > > > back
>> >> > > > > > from zero to 125.
>> >> > > > > >
>> >> > > > > > Some options to address this:
>> >> > > > > >
>> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
>> >> > > > > > and 2.x: This will further slow down the pre-commit run.
>> >> > > > > >
>> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
>> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
>> >> failures
>> >> > > > > > won't get ignored.
>> >> > > > > >
>> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test
>> Hive
>> >> 2.x
>> >> > > on
>> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit
>> tests.
>> >> > > > > > The pros are:
>> >> > > > > > - Simple test matrix.
>> >> > > > > > - No need to slow down test run or maintain version specific
>> >> golden
>> >> > > > files
>> >> > > > > > (**).
>> >> > > > > > - 2.x looks like the logical place to move to JDK8.
>> >> > > > > > - User

[jira] [Created] (HIVE-13883) WebHCat leaves token crc file never gets deleted

2016-05-27 Thread niklaus xiao (JIRA)

niklaus xiao created HIVE-13883:
---

 Summary: WebHCat leaves token crc file never gets deleted
 Key: HIVE-13883
 URL: https://issues.apache.org/jira/browse/HIVE-13883
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 2.0.1, 1.2.1, 1.1.1, 1.2.0
Reporter: niklaus xiao
Priority: Minor


In one of our long run environment, there are thousands of 
/tmp/.templeton*.tmp.crc files, 

{quote}
omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc 
...
-rw-r--r-- 1 omm  wheel 12 May 26 18:15 
/tmp/.templeton6676048390600607654.tmp.crc
-rw-r--r-- 1 omm  wheel 12 May 26 18:14 
/tmp/.templeton2733383617337556503.tmp.crc
-rw-r--r-- 1 omm  wheel 12 May 26 18:12 
/tmp/.templeton2183121761801669064.tmp.crc
-rw-r--r-- 1 omm  wheel 12 May 26 18:11 
/tmp/.templeton2689764046140543879.tmp.crc
...
{quote}

{quote}
omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc  | wc -l
17986
{quote}

It's created by webhcat, 
[https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
  and never gets deleted 
[https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

2016-05-27 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/
---

(Updated May 28, 2016, 2:04 a.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a404bd 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 cffa493 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
5ab7b3c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
fea3dc7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 b3b571d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java
 PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 e0f0676 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
ef2b7f7 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
4a96355 

Diff: https://reviews.apache.org/r/46754/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: [VOTE] Apache Hive 2.0.1 Release Candidate 1

2016-05-27 Thread Sergey Shelukhin

Restaged into 
https://repository.apache.org/content/repositories/orgapachehive-1051/.
I will release after the long weekend if there are no objections.

On 16/5/26, 14:11, "Owen O'Malley"  wrote:

>We don't vote on the artifacts in Nexus, but the source release. We trust
>you to recreate the jars in Nexus accurately based on the tags & tarballs.
>
>.. Owen
>
>On Thu, May 26, 2016 at 12:22 PM, Sergey Shelukhin
>
>wrote:
>
>> Unfortunately, the Nexus repo has expired while the vote was in
>>progress.
>> We will have to re-stage.
>> I assume people just downloaded the artifacts from the web link to vote…
>> How should we proceed? I can restage branch-2 as of the tag, should we
>> have a repeat vote after that where someone actually tries the repo? ;)
>>
>> On 16/5/24, 10:40, "Sergey Shelukhin"  wrote:
>>
>> >With 4 votes the vote passes. I am waiting for the release to
>>propagate to
>> >mirrors before doing the remaining steps.
>> >
>> >On 16/5/24, 10:38, "Owen O'Malley"  wrote:
>> >
>> >>+1
>> >>* Verified the signatures
>> >>* Signed Sergey's signature (and pushed to the keyservers)
>> >>
>> >>.. Owen
>> >>
>> >>
>> >>On Mon, May 23, 2016 at 11:50 PM, Thejas Nair 
>> >>wrote:
>> >>
>> >>> +1
>> >>> Also checked the release notes updates.
>> >>>
>> >>>
>> >>> On Thu, May 12, 2016 at 12:28 PM, Alan Gates 
>> >>>wrote:
>> >>> > +1.  Checked the keys and signatures, did a full build with a
>>clean
>> >>> repo, checked for the presence of any binary files, checked the
>>NOTICE
>> >>>and
>> >>> LICENSE files.
>> >>> >
>> >>> > Alan.
>> >>> >
>> >>> >> On May 3, 2016, at 21:25, Sergey Shelukhin 
>> >>>wrote:
>> >>> >>
>> >>> >> Apache Hive 2.0.1 Release Candidate 1 is available here:
>> >>> >>
>> >>> >> http://home.apache.org/~sershe/hive-2.0.1-rc1/
>> >>> >>
>> >>> >>
>> >>> >> Maven artifacts are available here:
>> >>> >>
>> >>> >>
>> 
>https://repository.apache.org/content/repositories/orgapachehive-1049/
>> >>> >>
>> >>> >>
>> >>> >> Source tag for RC1 (github mirror) is:
>> >>> >> https://github.com/apache/hive/releases/tag/release-2.0.1-rc1
>> >>> >>
>> >>> >>
>> >>> >> Voting will conclude in 72 hours.
>> >>> >>
>> >>> >> Hive PMC Members: Please test and vote.
>> >>> >>
>> >>> >> Thanks.
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >
>> >>>
>> >
>>
>>

Re: Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

2016-05-27 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47546/
---

(Updated May 28, 2016, 12:14 a.m.)


Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description (updated)
---

.


Diffs (updated)
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 cffa493 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 

Diff: https://reviews.apache.org/r/47546/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 47916: HIVE-13857 insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

2016-05-27 Thread Hari Sankar Sivarama Subramaniyan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47916/
---

(Updated May 27, 2016, 9:50 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

insert overwrite select from some table fails throwing 
org.apache.hadoop.security.AccessControlException - II


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bc39994 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 379eddc 
  shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java c2060fc 

Diff: https://reviews.apache.org/r/47916/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-05-27 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

(Updated May 27, 2016, 9:32 p.m.)


Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a404bd 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
349ee14 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinatorImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
 PRE-CREATION 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
83d492a 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 47916: HIVE-13857 insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

2016-05-27 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47916/#review135325
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (lines 2867 - 2876)


I don't think this is needed. This codepath gets exercised when dest is 
pre-existing. There is no need to set permissions in that case.


- Ashutosh Chauhan


On May 27, 2016, 7:42 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47916/
> ---
> 
> (Updated May 27, 2016, 7:42 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> insert overwrite select from some table fails throwing 
> org.apache.hadoop.security.AccessControlException - II
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 717589a 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 3fa1233 
>   shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java c2060fc 
> 
> Diff: https://reviews.apache.org/r/47916/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"

2016-05-27 Thread Aihua Xu (JIRA)

Aihua Xu created HIVE-13882:
---

 Summary: When hive.server2.async.exec.async.compile is turned on, 
from JDBC we will get "The query did not generate a result set" 
 Key: HIVE-13882
 URL: https://issues.apache.org/jira/browse/HIVE-13882
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 2.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu


 The following would fail with  "The query did not generate a result set"

stmt.execute("SET hive.driver.parallel.compilation=true");
stmt.execute("SET hive.server2.async.exec.async.compile=true");
ResultSet res =  stmt.executeQuery("SELECT * FROM " + tableName);
res.next();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [discuss] jdk8 support

2016-05-27 Thread Sergio Pena

Thanks Mohit.

I will plan to do the JDK8 change on Jenkins today EOD and monitor any
issues through the weekend.

- Sergio


On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal  wrote:

> Update on moving Hive2 tests to JDK8: I've addressed almost all test
> failures in HIVE-13547 on java8 branch. There is one remaining
> open item (HIVE-13834) which is currently assigned. Given current
> state of flaky test runs, there might be few more.
>
> I will work with Sergio to merge the test fixes to master and switch
> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
>
> After Hive2 tests switch, if your patch sees ordering related test
> failures in pre-commit run, it's likely JDK version related and you'll
> need to build & re-run the test using JDK8. Number of such tests
> should be relatively small.
>
> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal 
> wrote:
>
> > Created HIVE-13547 to track switching 2x tests to JDK8.
> >
> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena 
> > wrote:
> >
> >> I agree with such change as JDK7 is not longer supported.
> >>
> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
> >> replace the path from java7 to java8. But I think we should fix all JDK8
> >> issues or most of them before doing the change or we will end up having
> a
> >> lot of failures on all JIRAs running pre-commit tests.
> >>
> >> +1 with the change.
> >>
> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth 
> wrote:
> >>
> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> >> requirement
> >> > soon as well.
> >> >
> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho 
> wrote:
> >> >
> >> > > Sounds like a good plan, +1
> >> > >
> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> mo...@cloudera.com>
> >> > > wrote:
> >> > >
> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
> >> > since
> >> > > > we're not using any JDK8 specific features yet. So, option (3)
> >> > > essentially
> >> > > > just means using JDK8 exclusively to build & test Hive2.
> >> > > >
> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> >> > hashut...@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Mohit,
> >> > > > >
> >> > > > > I also think option 3 makes sense. We should strive to keep test
> >> > matrix
> >> > > > > small so that we can do fast QA runs for dev patches.
> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build &
> >> test
> >> > > > Hive2.
> >> > > > > However, I am not sure of explicitly dropping support altogether
> >> for
> >> > > jdk7
> >> > > > > on Hive2. We should make sure that in pom.xml java source &
> target
> >> > > > > compatibility is still 1.7 (which already is the case currently)
> >> so
> >> > > that
> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we can
> >> run
> >> > on
> >> > > > > jdk8.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Ashutosh
> >> > > > >
> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> >> mo...@cloudera.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi folks,
> >> > > > > >
> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> >> > > > > > In order to support Java 8, we fixed quite a bunch of tests in
> >> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
> >> tests
> >> > > run
> >> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
> >> > > result,
> >> > > > > > the count has crept up
> >> > > > > > <
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> >> > > > > > >
> >> > > > > > back
> >> > > > > > from zero to 125.
> >> > > > > >
> >> > > > > > Some options to address this:
> >> > > > > >
> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> >> > > > > > and 2.x: This will further slow down the pre-commit run.
> >> > > > > >
> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
> >> failures
> >> > > > > > won't get ignored.
> >> > > > > >
> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test
> Hive
> >> 2.x
> >> > > on
> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit
> tests.
> >> > > > > > The pros are:
> >> > > > > > - Simple test matrix.
> >> > > > > > - No need to slow down test run or maintain version specific
> >> golden
> >> > > > files
> >> > > > > > (**).
> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> >> > > > > > - Users transitioning to JDK8 for all other services do not
> >> have to
> >> > > > > > maintain
> >> > > > > > multiple java versions on the cluster.
> >> > > > > >
> >> > > > > > Option (3) looks most attractive to me.
> >> > > > > >
> >> > > > > > Moving to JDK8 also lines us up better for Java 9, which is on
> >> the
> >> > > > > > hor

Re: Review Request 47916: HIVE-13857 insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

2016-05-27 Thread Hari Sankar Sivarama Subramaniyan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47916/
---

(Updated May 27, 2016, 7:42 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

insert overwrite select from some table fails throwing 
org.apache.hadoop.security.AccessControlException - II


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 717589a 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 3fa1233 
  shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java c2060fc 

Diff: https://reviews.apache.org/r/47916/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-13881) Update curator version

2016-05-27 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-13881:
---

 Summary: Update curator version
 Key: HIVE-13881
 URL: https://issues.apache.org/jira/browse/HIVE-13881
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13880) add HiveAuthzContext to grant/revoke methods in HiveAuthorizer api

2016-05-27 Thread Thejas M Nair (JIRA)

Thejas M Nair created HIVE-13880:


 Summary: add HiveAuthzContext to grant/revoke methods in 
HiveAuthorizer api
 Key: HIVE-13880
 URL: https://issues.apache.org/jira/browse/HIVE-13880
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair


Passing context information to grant/revoke methods will help auditing logging 
those methods by authorizer plugin implementations.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13879) add HiveAuthzContext to grant/revoke methods in HiveAuthorizer api

2016-05-27 Thread Thejas M Nair (JIRA)

Thejas M Nair created HIVE-13879:


 Summary: add HiveAuthzContext to grant/revoke methods in 
HiveAuthorizer api
 Key: HIVE-13879
 URL: https://issues.apache.org/jira/browse/HIVE-13879
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair


Passing context information to grant/revoke methods will help auditing logging 
those methods by authorizer plugin implementations.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hive pull request: Hive 13867

2016-05-27 Thread thejasmn

GitHub user thejasmn opened a pull request:

https://github.com/apache/hive/pull/79

Hive 13867



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thejasmn/hive HIVE-13867

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/79.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #79


commit fe1ad0913d43964e72c924c1e89c175214cee807
Author: Thejas Nair 
Date:   2016-05-27T05:54:29Z

restore getUserIpAddress in QueryContext

commit a5906b36f037d976126db7da58af39828e8e96b7
Author: Thejas Nair 
Date:   2016-05-27T06:15:04Z

restore test

commit f00102fd2007bc213484827bfaba177291e164d0
Author: Thejas Nair 
Date:   2016-05-27T18:06:48Z

rename QueryContext.java to HiveAuthzContext.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Fwd: How is LOCATION used in CREATE INDEX ?

2016-05-27 Thread Sergio Pena

Hi All,

Anyone knows what is the use of the LOCATION keyword when creating an index?

There is an issue with CREATE INDEX when we specify the LOCATION to a table
location, and then rebuild the index. It writes unwanted data to it.

Also, if we use '/user/hive/warehouse' in the LOCATION, and then we DROP
INDEX, then the location is deleted.

I haven't found if the LOCATION is to store indexes data, or it is a table
location. When looking at the AST, a TOK_TABLELOCATION is added. This seems
like a table location, right?


hive> select * from encrypted;
1   sergio

hive> create index enc_index on table encrypted(id) as
'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' with deferred
rebuild location '/user/hive/warehouse/encrypted';

hive> alter index enc_index on encrypted rebuild;

hive> select * from encrypted;
1   hdfs://localhost:9000/user/hive/warehouse/encrypted/00_0

Thanks,
- Sergio

Review Request 47966: HIVE-13807

2016-05-27 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47966/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13807
https://issues.apache.org/jira/browse/HIVE-13807


Repository: hive-git


Description
---

HIVE-13807


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
 85e66d55604c78209b963d496e8d3d03c428f225 
  ql/src/test/results/clientpositive/input26.q.out 
74035c565597c2e8e0e8ef878a2e1292dbe2cf76 
  ql/src/test/results/clientpositive/llap/tez_union_dynamic_partition.q.out 
ce866400106e667d98affbbaf677f5f207dd48b5 
  ql/src/test/results/clientpositive/tez/tez_union_dynamic_partition.q.out 
a1221a4a954e993355df6502e6a046b852609113 
  ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out 
5e166e64da105ea05e1c14edd50bdee763758b86 

Diff: https://reviews.apache.org/r/47966/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-27 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review135251
---


Ship it!




Looks good. Could you please add a timeout annotation on the test before 
committing.

- Siddharth Seth


On May 27, 2016, 12:39 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 27, 2016, 12:39 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a404bd 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
>  ebc91b1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  7c47f0b 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 5ab7b3c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  04df929 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  c55436b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  e0f0676 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13878) Vectorization: Column pruning for Text vectorization

2016-05-27 Thread Gopal V (JIRA)

Gopal V created HIVE-13878:
--

 Summary: Vectorization: Column pruning for Text vectorization
 Key: HIVE-13878
 URL: https://issues.apache.org/jira/browse/HIVE-13878
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 2.1.0
Reporter: Gopal V
Assignee: Gopal V


Column pruning in TextFile vectorization does not work with Vector SerDe 
settings due to LazySimple deser codepath issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47838: HIVE-13831

2016-05-27 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47838/
---

(Updated May 27, 2016, 1:14 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13831
https://issues.apache.org/jira/browse/HIVE-13831


Repository: hive-git


Description
---

HIVE-13831


Diffs (updated)
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java
 d5cc9a5491edddc4fae840e087b384587ac17ddc 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PushdownTuple.java
 f326d522daf694a788b7b52f23da4134bc4251e6 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java
 18fb5eaebd87f2f194152241b3444adc3e24c5c3 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
42ea24e41883d8c02b49848020a80fcc7f179c50 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out 
590906b63e18c64cbadc5b02d978cbad41c8220f 
  hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
a42e36f0132457d22240d3ef4b8a52375a14fe34 
  hbase-handler/src/test/results/positive/hbase_timestamp.q.out 
29a04f9167803c896f998ed75a89e8f20ef4799e 
  hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 
34c3b230bd64f4fe9aa92a0e874e0f0a4919d208 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
2f0deae34ac42e5f6cbfffc6148473b62fb82d0c 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 
3a2ecb7282a2eed0b1f7e49ca70b15f10c8e9675 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 
7d7c764f8d800d90c7b23ed61f7deb589e51627d 

Diff: https://reviews.apache.org/r/47838/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 47838: HIVE-13831

2016-05-27 Thread Jesús Camacho Rodríguez



> On May 26, 2016, 8:08 p.m., Ashutosh Chauhan wrote:
> > I think we need to make this fix only when 
> > HBaseSerde.HBASE_TABLE_DEFAULT_STORAGE_TYPE='string', if its binary.. we 
> > can push these predicates since then lexicographical comparison would be 
> > correct.
> 
> Jesús Camacho Rodríguez wrote:
> You are right, I did not think about this. I have done it in the new 
> patch.

I am retracting on the above; I checked with the patch of HIVE-13693 on top, 
and even for binary representation, open range filters do not work. Maybe even 
if it is the binary representation of the value, it is ending up being stored 
as a String in HBase. This needs further investigation, but since currently it 
is not working properly in any case, I think for 2.1.0 we could just restrict 
binary as well as string; then HIVE-13693 can go in, and we can explore this 
issue further in the future.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47838/#review135034
---


On May 27, 2016, 12:40 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47838/
> ---
> 
> (Updated May 27, 2016, 12:40 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13831
> https://issues.apache.org/jira/browse/HIVE-13831
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13831
> 
> 
> Diffs
> -
> 
>   
> accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java
>  d5cc9a5491edddc4fae840e087b384587ac17ddc 
>   
> accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PushdownTuple.java
>  f326d522daf694a788b7b52f23da4134bc4251e6 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java
>  18fb5eaebd87f2f194152241b3444adc3e24c5c3 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 42ea24e41883d8c02b49848020a80fcc7f179c50 
>   hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
> a42e36f0132457d22240d3ef4b8a52375a14fe34 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out 
> 29a04f9167803c896f998ed75a89e8f20ef4799e 
>   hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 
> 34c3b230bd64f4fe9aa92a0e874e0f0a4919d208 
>   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
> 2f0deae34ac42e5f6cbfffc6148473b62fb82d0c 
>   ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 
> 3a2ecb7282a2eed0b1f7e49ca70b15f10c8e9675 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java
>  7d7c764f8d800d90c7b23ed61f7deb589e51627d 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 5390ba74aa40e1686d151f867dd74553aa07a1f1 
> 
> Diff: https://reviews.apache.org/r/47838/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 47838: HIVE-13831

2016-05-27 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47838/
---

(Updated May 27, 2016, 12:40 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13831
https://issues.apache.org/jira/browse/HIVE-13831


Repository: hive-git


Description
---

HIVE-13831


Diffs (updated)
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java
 d5cc9a5491edddc4fae840e087b384587ac17ddc 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PushdownTuple.java
 f326d522daf694a788b7b52f23da4134bc4251e6 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java
 18fb5eaebd87f2f194152241b3444adc3e24c5c3 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
42ea24e41883d8c02b49848020a80fcc7f179c50 
  hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
a42e36f0132457d22240d3ef4b8a52375a14fe34 
  hbase-handler/src/test/results/positive/hbase_timestamp.q.out 
29a04f9167803c896f998ed75a89e8f20ef4799e 
  hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 
34c3b230bd64f4fe9aa92a0e874e0f0a4919d208 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
2f0deae34ac42e5f6cbfffc6148473b62fb82d0c 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 
3a2ecb7282a2eed0b1f7e49ca70b15f10c8e9675 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 
7d7c764f8d800d90c7b23ed61f7deb589e51627d 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
5390ba74aa40e1686d151f867dd74553aa07a1f1 

Diff: https://reviews.apache.org/r/47838/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 47838: HIVE-13831

2016-05-27 Thread Jesús Camacho Rodríguez



> On May 26, 2016, 8:08 p.m., Ashutosh Chauhan wrote:
> > I think we need to make this fix only when 
> > HBaseSerde.HBASE_TABLE_DEFAULT_STORAGE_TYPE='string', if its binary.. we 
> > can push these predicates since then lexicographical comparison would be 
> > correct.

You are right, I did not think about this. I have done it in the new patch.


> On May 26, 2016, 8:08 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java,
> >  line 601
> > 
> >
> > I think you may use single boolean variable here and name it as 
> > nonEqualityPredicate

We need to check whether it is e.g. key < 5 and key > 2, as we might have key < 
2 and key < 5.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47838/#review135034
---


On May 25, 2016, 5:59 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47838/
> ---
> 
> (Updated May 25, 2016, 5:59 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13831
> https://issues.apache.org/jira/browse/HIVE-13831
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13831
> 
> 
> Diffs
> -
> 
>   
> accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java
>  d5cc9a5491edddc4fae840e087b384587ac17ddc 
>   
> accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PushdownTuple.java
>  f326d522daf694a788b7b52f23da4134bc4251e6 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 42ea24e41883d8c02b49848020a80fcc7f179c50 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out 
> 590906b63e18c64cbadc5b02d978cbad41c8220f 
>   hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
> a42e36f0132457d22240d3ef4b8a52375a14fe34 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out 
> 29a04f9167803c896f998ed75a89e8f20ef4799e 
>   hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 
> 34c3b230bd64f4fe9aa92a0e874e0f0a4919d208 
>   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
> 2f0deae34ac42e5f6cbfffc6148473b62fb82d0c 
>   ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 
> 3a2ecb7282a2eed0b1f7e49ca70b15f10c8e9675 
> 
> Diff: https://reviews.apache.org/r/47838/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Review Request 47948: HIVE-13808

2016-05-27 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47948/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13808
https://issues.apache.org/jira/browse/HIVE-13808


Repository: hive-git


Description
---

HIVE-13808


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
265dd7e61a192452f3910ad4937f4db1fa491dc3 
  ql/src/test/results/clientpositive/bucket_groupby.q.out 
1afab38ab41143c2326fefc64525fd51839918f7 
  ql/src/test/results/clientpositive/cp_sel.q.out 
9ee9754ed17548f97476c782bd12ac186e8a213f 
  ql/src/test/results/clientpositive/decimal_stats.q.out 
72806f0e9cab1272195d781af43ed96b65e8c805 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 
86346b3492f216418075c1483c8941c0a1f28c77 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
ac95ec2a12b6f12dd08c1b9a51135531da0fb526 
  ql/src/test/results/clientpositive/pcr.q.out 
9daddfb5d9070273a2e3782df7ccd3835060dc87 
  ql/src/test/results/clientpositive/perf/query31.q.out 
935631c79ceae8c2c2a1add8a7db7b986ac7c7c9 
  ql/src/test/results/clientpositive/perf/query64.q.out 
5a5ee74fd5b2bea113a8de20fb277587af76cb47 
  ql/src/test/results/clientpositive/perf/query66.q.out 
5eef7e70039679e456d1efdfb4963a569972fa6f 
  ql/src/test/results/clientpositive/perf/query75.q.out 
802c1af45126753551cda573583502f4a70170f7 
  ql/src/test/results/clientpositive/pointlookup2.q.out 
5facf9af8cc740a20c35fba3505fe0b2858bc6cf 
  ql/src/test/results/clientpositive/pointlookup3.q.out 
def1bc841d2bfb5ae0afe895f25e5e6ffaff5b5f 
  ql/src/test/results/clientpositive/ppd_udf_case.q.out 
56941cde128a7813ae60f0dac0edde2830a01c9f 
  ql/src/test/results/clientpositive/smb_mapjoin_18.q.out 
4b29056379f9045095de326a03571a87f766030c 
  ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 
86c5bd759d7e220ebdd44c2aee88a62297d2c19e 
  ql/src/test/results/clientpositive/spark/pcr.q.out 
cd16787c2eb18095a00dcf3f0be173d8c32c1dca 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_18.q.out 
bfdd529fa08e56681efc6366cf57916bbffe89d2 
  ql/src/test/results/clientpositive/spark/union_view.q.out 
7ac641f7ceee1744a4d25864940669076817004b 
  ql/src/test/results/clientpositive/tez/vector_coalesce.q.out 
e705e606faee2b081a4ea8932ea70bf290e604c2 
  ql/src/test/results/clientpositive/tez/vector_date_1.q.out 
9f31820dee579e0fe650bc59dbd68a259d7fec1f 
  ql/src/test/results/clientpositive/tez/vector_decimal_2.q.out 
292fb8892909cd3b8e2025f4b5f5d58925de8292 
  ql/src/test/results/clientpositive/tez/vector_decimal_round_2.q.out 
3b6d42ddd507c7c096a6424178fd08bb7c036fcf 
  ql/src/test/results/clientpositive/tez/vector_interval_arithmetic.q.out 
064e319d13d7a1631b4fda5621257c5093fcd5e9 
  ql/src/test/results/clientpositive/union_view.q.out 
90e240a0021ed5e58b6dc4fe1e299c702e7e503b 
  ql/src/test/results/clientpositive/vector_coalesce.q.out 
21a91229bdd64a0f659deda4b3ca06f3c1253878 
  ql/src/test/results/clientpositive/vector_date_1.q.out 
a394e0f6d12eb45218518c42131fff42ae3fce22 
  ql/src/test/results/clientpositive/vector_decimal_2.q.out 
892a1b3fa51056a9a1c378b693bc63d1efda7d60 
  ql/src/test/results/clientpositive/vector_decimal_round_2.q.out 
5b321c2b3e0acadd9ef7fa14eb7f0dfbdce34385 
  ql/src/test/results/clientpositive/vector_interval_1.q.out 
379747c3d54a23c2d5cd74582e557fa47fa7f4b2 
  ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out 
bad7e4a827066e33bb5f470b19e80fbea1822495 

Diff: https://reviews.apache.org/r/47948/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-13877) Hplsql UDF doesn't work in Hive Cli

2016-05-27 Thread jiangxintong (JIRA)

jiangxintong created HIVE-13877:
---

 Summary: Hplsql UDF doesn't work in Hive Cli
 Key: HIVE-13877
 URL: https://issues.apache.org/jira/browse/HIVE-13877
 Project: Hive
  Issue Type: Bug
  Components: hpl/sql
Affects Versions: 2.2.0
Reporter: jiangxintong


Hive cli will throw "Error evaluating hplsql" exception when i use the hplsql 
udf like "SELECT hplsql('hello[:1]', columnName) FROM tableName".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13876) Vectorization: Port HIVE-11544 to LazySimpleDeserializeRead

2016-05-27 Thread Gopal V (JIRA)

Gopal V created HIVE-13876:
--

 Summary: Vectorization: Port HIVE-11544 to 
LazySimpleDeserializeRead
 Key: HIVE-13876
 URL: https://issues.apache.org/jira/browse/HIVE-13876
 Project: Hive
  Issue Type: Bug
  Components: File Formats, Vectorization
Affects Versions: 2.1.0
Reporter: Gopal V






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hive pull request: Hplsql udf doesn't work in hive cli

2016-05-27 Thread jiangxt2

GitHub user jiangxt2 opened a pull request:

https://github.com/apache/hive/pull/78

Hplsql udf doesn't work in hive cli

Hive cli will throw "Error evaluating hplsql" exception when i use the 
hplsql udf like "SELECT hplsql('hello[:1]', columnName) FROM tableName". So, 
this request exists to fix that problem.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiangxt2/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/78.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #78


commit 99d4c032d8746db6346585c9cece0bb1a0cc1458
Author: jiangxt2 
Date:   2016-05-27T08:40:52Z

Hplsql udf doesn't work in hive cli




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-13875) Beeline ignore where clause when it is the last line of file and missing a EOL hence give wrong query result

2016-05-27 Thread Lu Ji (JIRA)

Lu Ji created HIVE-13875:


 Summary: Beeline ignore where clause when it is the last line of 
file and missing a EOL hence give wrong query result
 Key: HIVE-13875
 URL: https://issues.apache.org/jira/browse/HIVE-13875
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 1.2.1
Reporter: Lu Ji
Priority: Minor


Steps to reproduce:
Say we have a simple table:
{code}
select * from lji.lu_test;
+---+--+--+
| lu_test.name  | lu_test.country  |
+---+--+--+
| john  | us   |
| hong  | cn   |
+---+--+--+
2 rows selected (0.04 seconds)
{code}
We have a simple query in a file. But note this file missing the last EOL.
{code}
[lji@hqetghdped90 ~]$ cat -A test.hql
use lji;$
select * from lu_test$
where country='us';[lji@hqetghdped90 ~]$
{code}
Then if we execute file using both hive CLI and beeline + HS2, we have 
different result.
{code}
[lji@hqetghdped90 ~]$ hive -f test.hql
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in 
file:/etc/hive/2.3.4.7-4/0/hive-log4j.properties
OK
Time taken: 1.624 seconds
OK
johnus
Time taken: 1.482 seconds, Fetched: 1 row(s)
[lji@hqetghdped90 ~]$ beeline -u 
"jdbc:hive2://hqetghdped90.pclc0.merkle.local:1/default;principal=hive/_HOST@PCLC0.MERKLE.LOCAL"
 -f test.hql
WARNING: Use "yarn jar" to launch YARN applications.
Connecting to 
jdbc:hive2://hqetghdped90.pclc0.merkle.local:1/default;principal=hive/_HOST@PCLC0.MERKLE.LOCAL
Connected to: Apache Hive (version 1.2.1.2.3.4.7-4)
Driver: Hive JDBC (version 1.2.1.2.3.4.7-4)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hqetghdped90.pclc0.merkle.loc> use lji;
No rows affected (0.06 seconds)
0: jdbc:hive2://hqetghdped90.pclc0.merkle.loc> select * from lu_test
0: jdbc:hive2://hqetghdped90.pclc0.merkle.loc> where 
country='us';+---+--+--+
| lu_test.name  | lu_test.country  |
+---+--+--+
| john  | us   |
| hong  | cn   |
+---+--+--+
2 rows selected (0.073 seconds)
0: jdbc:hive2://hqetghdped90.pclc0.merkle.loc>
Closing: 0: 
jdbc:hive2://hqetghdped90.pclc0.merkle.local:1/default;principal=hive/_HOST@PCLC0.MERKLE.LOCAL
{code}
Obviously, beeline gave the wrong result. It ignore the where clause in the 
last line.
I know it is quit weird for a file missing the last EOL, but for whatever 
reason, we kind of having quit some files in this state. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [discuss] jdk8 support

2016-05-27 Thread Mohit Sabharwal

Update on moving Hive2 tests to JDK8: I've addressed almost all test
failures in HIVE-13547 on java8 branch. There is one remaining
open item (HIVE-13834) which is currently assigned. Given current
state of flaky test runs, there might be few more.

I will work with Sergio to merge the test fixes to master and switch
the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.

After Hive2 tests switch, if your patch sees ordering related test
failures in pre-commit run, it's likely JDK version related and you'll
need to build & re-run the test using JDK8. Number of such tests
should be relatively small.

On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal 
wrote:

> Created HIVE-13547 to track switching 2x tests to JDK8.
>
> On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena 
> wrote:
>
>> I agree with such change as JDK7 is not longer supported.
>>
>> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
>> replace the path from java7 to java8. But I think we should fix all JDK8
>> issues or most of them before doing the change or we will end up having a
>> lot of failures on all JIRAs running pre-commit tests.
>>
>> +1 with the change.
>>
>> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth  wrote:
>>
>> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
>> requirement
>> > soon as well.
>> >
>> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho  wrote:
>> >
>> > > Sounds like a good plan, +1
>> > >
>> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal 
>> > > wrote:
>> > >
>> > > > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
>> > since
>> > > > we're not using any JDK8 specific features yet. So, option (3)
>> > > essentially
>> > > > just means using JDK8 exclusively to build & test Hive2.
>> > > >
>> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
>> > hashut...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Hi Mohit,
>> > > > >
>> > > > > I also think option 3 makes sense. We should strive to keep test
>> > matrix
>> > > > > small so that we can do fast QA runs for dev patches.
>> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build &
>> test
>> > > > Hive2.
>> > > > > However, I am not sure of explicitly dropping support altogether
>> for
>> > > jdk7
>> > > > > on Hive2. We should make sure that in pom.xml java source & target
>> > > > > compatibility is still 1.7 (which already is the case currently)
>> so
>> > > that
>> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we can
>> run
>> > on
>> > > > > jdk8.
>> > > > >
>> > > > > Thanks,
>> > > > > Ashutosh
>> > > > >
>> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
>> mo...@cloudera.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi folks,
>> > > > > >
>> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
>> > > > > > In order to support Java 8, we fixed quite a bunch of tests in
>> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
>> tests
>> > > run
>> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
>> > > result,
>> > > > > > the count has crept up
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
>> > > > > > >
>> > > > > > back
>> > > > > > from zero to 125.
>> > > > > >
>> > > > > > Some options to address this:
>> > > > > >
>> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
>> > > > > > and 2.x: This will further slow down the pre-commit run.
>> > > > > >
>> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
>> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
>> failures
>> > > > > > won't get ignored.
>> > > > > >
>> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive
>> 2.x
>> > > on
>> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
>> > > > > > The pros are:
>> > > > > > - Simple test matrix.
>> > > > > > - No need to slow down test run or maintain version specific
>> golden
>> > > > files
>> > > > > > (**).
>> > > > > > - 2.x looks like the logical place to move to JDK8.
>> > > > > > - Users transitioning to JDK8 for all other services do not
>> have to
>> > > > > > maintain
>> > > > > > multiple java versions on the cluster.
>> > > > > >
>> > > > > > Option (3) looks most attractive to me.
>> > > > > >
>> > > > > > Moving to JDK8 also lines us up better for Java 9, which is on
>> the
>> > > > > > horizon (Oracle will end public updates for Java 8 in Sep 2017)
>> > > > > >
>> > > > > > Around 100 of the latest crop of failures are due to one cause
>> > > > > > (HIVE-13409).
>> > > > > > I can take a pass at triaging the rest if there is consensus
>> around
>> > > > what
>> > > > > > to do overall.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mohit
>> > > > > >
>> > > > > > (*) Most test failures are due to JDK8 using a different hash
>> > > function
>> > > > > for
>> > >

[jira] [Created] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

Re: [discuss] jdk8 support

[jira] [Created] (HIVE-13883) WebHCat leaves token crc file never gets deleted

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

Re: [VOTE] Apache Hive 2.0.1 Release Candidate 1

Re: Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

Re: Review Request 47916: HIVE-13857 insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

Re: Review Request 47916: HIVE-13857 insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

[jira] [Created] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"

Re: [discuss] jdk8 support

Re: Review Request 47916: HIVE-13857 insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II

[jira] [Created] (HIVE-13881) Update curator version

[jira] [Created] (HIVE-13880) add HiveAuthzContext to grant/revoke methods in HiveAuthorizer api

[jira] [Created] (HIVE-13879) add HiveAuthzContext to grant/revoke methods in HiveAuthorizer api

[GitHub] hive pull request: Hive 13867

Fwd: How is LOCATION used in CREATE INDEX ?

Review Request 47966: HIVE-13807

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[jira] [Created] (HIVE-13878) Vectorization: Column pruning for Text vectorization

Re: Review Request 47838: HIVE-13831

Re: Review Request 47838: HIVE-13831

Re: Review Request 47838: HIVE-13831

Re: Review Request 47838: HIVE-13831

Review Request 47948: HIVE-13808

[jira] [Created] (HIVE-13877) Hplsql UDF doesn't work in Hive Cli

[jira] [Created] (HIVE-13876) Vectorization: Port HIVE-11544 to LazySimpleDeserializeRead

[GitHub] hive pull request: Hplsql udf doesn't work in hive cli

[jira] [Created] (HIVE-13875) Beeline ignore where clause when it is the last line of file and missing a EOL hence give wrong query result

Re: [discuss] jdk8 support

30 matches

Site Navigation

Mail list logo

Footer information