Re: [VOTE] Apache Hive 2.0.1 Release Candidate 1

[Thejas Nair]

+1
Also checked the release notes updates.


On Thu, May 12, 2016 at 12:28 PM, Alan Gates  wrote:
> +1.  Checked the keys and signatures, did a full build with a clean repo, 
> checked for the presence of any binary files, checked the NOTICE and LICENSE 
> files.
>
> Alan.
>
>> On May 3, 2016, at 21:25, Sergey Shelukhin  wrote:
>>
>> Apache Hive 2.0.1 Release Candidate 1 is available here:
>>
>> http://home.apache.org/~sershe/hive-2.0.1-rc1/
>>
>>
>> Maven artifacts are available here:
>>
>> https://repository.apache.org/content/repositories/orgapachehive-1049/
>>
>>
>> Source tag for RC1 (github mirror) is:
>> https://github.com/apache/hive/releases/tag/release-2.0.1-rc1
>>
>>
>> Voting will conclude in 72 hours.
>>
>> Hive PMC Members: Please test and vote.
>>
>> Thanks.
>>
>>
>>
>

[jira] [Created] (HIVE-13827) LLAPIF: authentication on the output channel

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-13827:
---

 Summary: LLAPIF: authentication on the output channel
 Key: HIVE-13827
 URL: https://issues.apache.org/jira/browse/HIVE-13827
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


The current thinking is that we'd send the token. There's no protocol on the 
channel right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13826) Make VectorUDFAdaptor work for GenericUDFBetween when used as FILTER

[Matt McCline (JIRA)]

Matt McCline created HIVE-13826:
---

 Summary: Make VectorUDFAdaptor work for GenericUDFBetween when 
used as FILTER
 Key: HIVE-13826
 URL: https://issues.apache.org/jira/browse/HIVE-13826
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


GenericUDFBetween doesn't vectorize with VectorUDFAdaptor when used as FILTER 
(i.e. as single item for WHERE).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized execution in IO

[Matt McCline]

> On May 23, 2016, 10:37 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java, line 181
> > 
> >
> > The VectorExtractRow class and extractRowColumn method has similar 
> > functionality.  But perhaps it is not a good reuse case.

Is this only being used for ORC which has already done Schema Evolution below?  
Or, will this be used for non-ORC tables?


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review134467
---


On May 23, 2016, 5:27 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> ---
> 
> (Updated May 23, 2016, 5:27 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  298f788 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079 
>   ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812 
>   ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION 
>   ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47735/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 41176: HIVE-12643

[Matt McCline]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41176/#review134468
---




ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java (line 88)


Good I see you are using the isInputFileFormatSelfDescribing method.


- Matt McCline


On May 18, 2016, 1:08 a.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41176/
> ---
> 
> (Updated May 18, 2016, 1:08 a.m.)
> 
> 
> Review request for hive, Gopal V and Matt McCline.
> 
> 
> Bugs: HIVE-12643
> https://issues.apache.org/jira/browse/HIVE-12643
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> For self describing InputFormat don't replicate schema information in 
> partitions
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 6bc882a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 449bef8 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 812af9a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 4a156a2 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 4d627ef 
>   ql/src/test/results/clientpositive/quotedid_tblproperty.q.out ca1dbe6 
>   ql/src/test/results/clientpositive/tez/vector_partition_diff_num_cols.q.out 
> f23a359 
>   ql/src/test/results/clientpositive/vector_partition_diff_num_cols.q.out 
> ef92b89 
> 
> Diff: https://reviews.apache.org/r/41176/diff/
> 
> 
> Testing
> ---
> 
> existing tests
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 47735: HIVE-13617 LLAP: support non-vectorized execution in IO

[Matt McCline]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/#review134467
---




ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java (line 181)


The VectorExtractRow class and extractRowColumn method has similar 
functionality.  But perhaps it is not a good reuse case.


- Matt McCline


On May 23, 2016, 5:27 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47735/
> ---
> 
> (Updated May 23, 2016, 5:27 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  298f788 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079 
>   ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812 
>   ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION 
>   ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47735/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/
---

(Updated May 23, 2016, 10:22 p.m.)


Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
 ebc91b1 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 7c47f0b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
5ab7b3c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 04df929 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
c55436b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 e0f0676 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 

Diff: https://reviews.apache.org/r/46956/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 290
> > 
> >
> > What user is expected over here.
> > 1. In case of an invocation by HS2 to run a Tez query - I'm assuming 
> > this would be the HS2 service user (which is the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.
> 
> Sergey Shelukhin wrote:
> This is the calling user in case of RPC.
> 
> Siddharth Seth wrote:
> This goes back to whether it's invoked locally or remotely.
> Local and Remote calls by HS2 to obtain a token on behalf of an external 
> client, will need to pass in the user name to generate the token correctly. 
> What's obtained on the RPC call will almost always be the Hive user - at 
> least for the remote call.
> Behaviour should not change if the flag to get the token locally / 
> remotely is changed.
> 
> Sergey Shelukhin wrote:
> This is protobuf, it's always invoked remotely. Two paths only meet at 
> SecretManager level.
> 
> Siddharth Seth wrote:
> Reference was to the way the token is obtained (locally or remotely) - 
> not the specific call.

fixed by only retaining the remote API for CLI and removing the setting. HS2 
user should have access to ZK paths, for registry if nothing else, so this 
shouldn't be a problem


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java,
> >  line 61
> > 
> >
> > I don't think we need to create a new instance of the 
> > ZKDelegationTokenSecretManager.
> > 
> > The one created earlier to generate tokens should be passed in.
> > 
> > The KeySigner could be an interface instead, and SecretManager (extends 
> > ZKDelegationTokenSecretManager) can implement this. ACL checks etc are 
> > already setup there. There's no requirement to have two independent copies 
> > of the ZKSecretManager running in the same daemon.
> 
> Sergey Shelukhin wrote:
> This one has completely different logic and even different template 
> parameter.
> 
> Siddharth Seth wrote:
> Most of the logic is in the BaseSecretManager itself, correct ?
> The same instance can be used to generate tokens, as well as sign. Is 
> there a downside to that ?
> Setting up two instances would create extra threads, and confusion while 
> debugging; Also potentially additional load to ZK, additional logins, etc

Merged the classes. They are created separately for now, I'll add central 
creation in one of the subsequent patches that adds llapcoordinator.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/a

[jira] [Created] (HIVE-13825) Map joins with cloned tables with same locations, but different column names throw error exceptions

[JIRA]

Sergio Peña created HIVE-13825:
--

 Summary: Map joins with cloned tables with same locations, but 
different column names throw error exceptions
 Key: HIVE-13825
 URL: https://issues.apache.org/jira/browse/HIVE-13825
 Project: Hive
  Issue Type: Bug
Reporter: Sergio Peña


The following scenario of 2 tables with same locations cannot be used on a JOIN 
query:

{noformat}
hive> create table t1 (a string, b string) location 
'/user/hive/warehouse/test1';
OK
hive> create table t2 (c string, d string) location 
'/user/hive/warehouse/test1';
OK
hive> select t1.a from t1 join t2 on t1.a = t2.c;
...
2016-05-23 16:39:57 Starting to launch local task to process map join;  
maximum memory = 477102080
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-4

Logs:

FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
{noformat}

The logs contain this error exception:
{noformat}
2016-05-23T16:39:58,163 ERROR [main]: mr.MapredLocalTask (:()) - Hive Runtime 
Error: Map local work failed
java.lang.RuntimeException: cannot find field a from [0:c, 1:d]
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
at 
org.apache.hadoop.hive.serde2.BaseStructObjectInspector.getStructFieldRef(BaseStructObjectInspector.java:133)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:973)
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:999)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:75)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:499)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:403)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:383)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:751)
{noformat}






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13824) NOSUCHMethodFound org.fusesource.jansi.internal.Kernel32.GetConsoleOutputCP()I

[Ekta Paliwal (JIRA)]

Ekta Paliwal created HIVE-13824:
---

 Summary: NOSUCHMethodFound 
org.fusesource.jansi.internal.Kernel32.GetConsoleOutputCP()I
 Key: HIVE-13824
 URL: https://issues.apache.org/jira/browse/HIVE-13824
 Project: Hive
  Issue Type: Bug
  Components: Beeline, Hive
 Environment: WIndows 8, HADOOP 2.7, HIVE 1.2.1, SPARK 1.6.1
Reporter: Ekta Paliwal



0
down vote
favorite
I have been trying to install hive on windows. I have 64 bit windows 8 on which 
HADOOP and SPARK are running. I have

1.HADOOP_HOME
2.HIVE_HOME
3.SPARK_HOME
4.Platform
5.PATH
all these variables set up on my system. Also, I was getting these error before

Missing Hive Execution Jar: 
C:\hadoop1\hadoop-2.7.2\apache-hive-1.2.1-bin/lib/hive-exec-*.jar

I solved these error by editing the Hive file inside bin folder of HIVE. These 
errors are because of the forward slash"/" in environment variables in HIVE 
file. I replace them with "\" and those errors are gone. But now I am facing 
another problem. I am getting these error

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/C:/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/C:/hadoop2.7/hadoop-2.7.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Beeline version 1.6.1 by Apache Hive
Exception in thread "main" java.lang.NoSuchMethodError: 
org.fusesource.jansi.internal.Kernel32.GetConsoleOutputCP()I
at jline.WindowsTerminal.getConsoleOutputCodepage(WindowsTerminal.java:293)
at jline.WindowsTerminal.getOutputEncoding(WindowsTerminal.java:186)
at jline.console.ConsoleReader.(ConsoleReader.java:230)
at jline.console.ConsoleReader.(ConsoleReader.java:221)
at jline.console.ConsoleReader.(ConsoleReader.java:209)
at org.apache.hive.beeline.BeeLine.getConsoleReader(BeeLine.java:834)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:770)
at 
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

I have searched alot on these. Also I have posted these question on HIVE User 
mailing List but got no response. Please help me with this. Not even getting 
results when google this error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13823) Remove unnecessary log line in common join operator

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-13823:
-

 Summary: Remove unnecessary log line in common join operator
 Key: HIVE-13823
 URL: https://issues.apache.org/jira/browse/HIVE-13823
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Fix For: 2.1.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Siddharth Seth]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 290
> > 
> >
> > What user is expected over here.
> > 1. In case of an invocation by HS2 to run a Tez query - I'm assuming 
> > this would be the HS2 service user (which is the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.
> 
> Sergey Shelukhin wrote:
> This is the calling user in case of RPC.
> 
> Siddharth Seth wrote:
> This goes back to whether it's invoked locally or remotely.
> Local and Remote calls by HS2 to obtain a token on behalf of an external 
> client, will need to pass in the user name to generate the token correctly. 
> What's obtained on the RPC call will almost always be the Hive user - at 
> least for the remote call.
> Behaviour should not change if the flag to get the token locally / 
> remotely is changed.
> 
> Sergey Shelukhin wrote:
> This is protobuf, it's always invoked remotely. Two paths only meet at 
> SecretManager level.

Reference was to the way the token is obtained (locally or remotely) - not the 
specific call.


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 290
> > 
> >
> > What user is expected over here.
> > 1. In case of an invocation by HS2 to run a Tez query - I'm assuming 
> > this would be the HS2 service user (which is the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.
> 
> Sergey Shelukhin wrote:
> This is the calling user in case of RPC.
> 
> Siddharth Seth wrote:
> This goes back to whether it's invoked locally or remotely.
> Local and Remote calls by HS2 to obtain a token on behalf of an external 
> client, will need to pass in the user name to generate the token correctly. 
> What's obtained on the RPC call will almost always be the Hive user - at 
> least for the remote call.
> Behaviour should not change if the flag to get the token locally / 
> remotely is changed.

This is protobuf, it's always invoked remotely. Two paths only meet at 
SecretManager level.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/#review134432
---



Any possibility of simple unit tests ?


llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java
 (line 36)


Lets skip the reflection, and create a single UGI for now.

A follow up jira to create a UGI pool should take care of the potential 
single UGI perf issue.


- Siddharth Seth


On May 20, 2016, 1:49 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46754/
> ---
> 
> (Updated May 20, 2016, 1:49 a.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9cc8fbe 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  cffa493 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> fea3dc7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  b3b571d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java
>  PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
> 
> Diff: https://reviews.apache.org/r/46754/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134430
---



Still needs some tests.


llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 (line 79)


Default to TRUE. Alternately - see next comment.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 (line 145)


This should explicitly check for "false". Any other value should either 
default to true or thrown an error. Err on the side of being secure.


- Siddharth Seth


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Siddharth Seth]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java,
> >  line 134
> > 
> >
> > Can a second login be avoided. I'm guessing this is because the ZK 
> > principla may be different from the llap principla.
> > What was the reason for them to be different again ? (Especially w.r.t 
> > the SecretManager). Not sure if the fallback to using the llap principal 
> > and keytab will work if they have to be different.
> 
> Sergey Shelukhin wrote:
> The same principal didn't work on the test cluster I had for some reason 
> that I no longer remember :(

The logic does a login with ZK keytab, otherwise fallback to the one used by 
LLAP.
Either the LLAP keytab works - in which case the ZK login is unnecessary, OR 
the LLAP keytab does not work - in which case the fallback should not exist.

If this is an unknown, should we create a follow up jira to investigate and 
fix, and meanwhile add a comment in the code ?


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java,
> >  line 170
> > 
> >
> > Maybe move all of these checks into the RPC layers itself ... i.e. 
> > LlapServiceServerImpl. As early as possible.
> 
> Sergey Shelukhin wrote:
> The permissions are checked in calls by ContainerRunner. RPC right now 
> just propagates the request...

The suggestion is to move it up :)


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 287
> > 
> >
> > All of this logic should be invoked even when obtaining tokens from 
> > ZKSM directly.
> > 
> > Whether Tez is being used, or an external client - as long as HS2 is 
> > obtaining a token, it can do it directly from ZK. This code path is not 
> > likely to be exercised a lot.
> > Assuming that invocation (when it happens, and likely needs another 
> > jira) - will call in to LlapTokenLocalClient.createToken directly - and 
> > will send in isSigningRequired based on all of the same configs.
> > 
> > Would be better to move the logic out of this function in that case.
> > 
> > Maybe the config flag itself could be dropped. If Tez, no singing, if 
> > external - force signing.
> 
> Sergey Shelukhin wrote:
> this actually kind of orthogonal. This logic doesn't apply to when HS2 
> creates the token preciusely because HS2 knows whether it's creating the 
> token for Tez or external, so it can set the flag accordingly.
> When the method is called remotely, by default we always require signing, 
> but that can be disabled for CLI, or HS2 calling remotely (presumably under 
> the same user as LLAP).

I think we need a ManagementInterface - which has the local and remote 
implementation. The if/else block in the core (TezSessionState) is fairly ugly. 
All of the implementation details and logic for the individual cases can hide 
in there. Likely a separate jira ?


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 290
> > 
> >
> > What user is expected over here.
> > 1. In case of an invocation by HS2 to run a Tez query - I'm assuming 
> > this would be the HS2 service user (which is the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.
> 
> Sergey Shelukhin wrote:
> This is the calling user in case of RPC.

This goes back to whether it's invoked locally or remotely.
Local and Remote calls by HS2 to obtain a token on behalf of an external 
client, will need to pass in the user name to generate the token correctly. 
What's obtained on the RPC call will almost always be the Hive user - at least 
for the remote call.
Behaviour should not change if the flag to get the token locally / remotely is 
changed.


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java,
> >  line 61
> > 
> >
> > I don't think we need to create a new in

[jira] [Created] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-13822:


 Summary: TestPerfCliDriver throws warning in StatsSetupConst that  
JsonParser cannot parse COLUMN_STATS
 Key: HIVE-13822
 URL: https://issues.apache.org/jira/browse/HIVE-13822
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


Thanks to [~jcamachorodriguez] for uncovering this issue as part of HIVE-13269. 
StatsSetupConst.areColumnStatsUptoDate() is used to check whether stats are 
up-to-date.  In case of PerfCliDriver, ‘false’ (thus, not up-to-date) is 
returned and the following debug message in the logs:

{code}
In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in 
StatsSetupConst)
{code}

Looks like the issue started happening after HIVE-12261 went in. 

The fix would be to replace
{{COLUMN_STATS_ACCURATE,true}}
with
{{COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}}}
in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13821) OrcSplit groups all delta files together into a single split

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-13821:


 Summary: OrcSplit groups all delta files together into a single 
split
 Key: HIVE-13821
 URL: https://issues.apache.org/jira/browse/HIVE-13821
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
 Attachments: HIVE-13821.1.patch

HIVE-7428 had fix for worst case column projection size estimate. It was 
removed in HIVE-10397 to return file length but for ACID strategy file length 
is passed as 0. In worst case, this always return 0 and all files ends up in 
single split. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java,
> >  line 267
> > 
> >
> > A follow up jira may be to limit the age of keys.
> > i.e. if a keyId is older than a certain amount of time - fail the 
> > request. I'm not sure how ZKSecretManager rotates these keys, and when they 
> > are invalidated.
> > 
> > A user can potentially use an old (presumably compromsied key) to 
> > generate requests - which will be valid if keys are not rotated/aged.

https://issues.apache.org/jira/browse/HIVE-13820


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13820) LLAP: investigate SecretManager key aging; add configs/aging mechanism if necessary

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-13820:
---

 Summary: LLAP: investigate SecretManager key aging; add 
configs/aging mechanism if necessary
 Key: HIVE-13820
 URL: https://issues.apache.org/jira/browse/HIVE-13820
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


See HIVE-13444



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 47735: HIVE-13617 LLAP: support non-vectorized execution in IO

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 298f788 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079 
  ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812 
  ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION 
  ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/47735/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-13819) Read & eXecute permissions on Database allows to ALTER it.

[Alexandre Linte (JIRA)]

Alexandre Linte created HIVE-13819:
--

 Summary: Read & eXecute permissions on Database allows to ALTER it.
 Key: HIVE-13819
 URL: https://issues.apache.org/jira/browse/HIVE-13819
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 1.2.1
 Environment: Hadoop 2.7.2, Hive 1.2.1, Kerberos.
Reporter: Alexandre Linte


Hi,

As the owner of an Hive database I can modify the Hive database metadata 
whereas I only has the read and execute permission on the Hive database 
repository.
I was expected to not be able to modify these metadata.

Context:
- Hive database configured with the Storage Based Authorization strategy.
- Hive client authorization is disabled.
- Metastore side security is activated.

Permission configuration:
{noformat}
dr-x--x---   - hive9990hive9990 0 2016-05-20 17:10 
/path/to/hive/warehouse/p09990.db
{noformat}

ALTER command as hive9990 user:
{noformat}
hive (p09990)>  ALTER DATABASE p09990 SET DBPROPERTIES ('comment'='database 
altered');
OK
Time taken: 0.277 seconds
hive (p09990)> DESCRIBE DATABASE EXTENDED p09990;
OK
p09990  hdfs://path/to/hive/warehouse/p09990.dbhdfsUSER
{comment=database altered}
{noformat}

Configuration of hive-site.xml on the metastore:
{noformat}




 
  
  hive.security.authorization.enabled
  false
  enable or disable the Hive client authorization
  

  
  hive.security.metastore.authorization.manager
  
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
  authorization manager class name to be used in the metastore 
for authorization.
  The user defined authorization class should implement interface 
org.apache.hadoop.hive.ql.security.authorization.HiveMetastoreAuthorizationProvider.
  
  

  
  hive.metastore.pre.event.listeners
  
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
  This turns on metastore-side security.
  
  

  
  hive.security.metastore.authorization.auth.reads
  true
  If this is true, the metastore authorizer authorizes read 
actions on database and table.
  
  

  
  hive.security.authorization.manager
  
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
  The Hive client authorization manager class name.
  The user defined authorization class should implement interface 
org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider.
  
  

  
  hive.security.authorization.createtable.owner.grants
  ALL
  the privileges automatically granted to the owner whenever a 
table gets created. 
   An example like "select,drop" will grant select and drop privilege to 
the owner of the table
  

  
  hive.users.in.admin.role
  hdfs
  Comma separated list of users who are in admin role for 
bootstrapping.
More users can be added in ADMIN role later.
  

  
  hive.metastore.warehouse.dir
  /path/to/hive/warehouse/
  location of default database for the warehouse
  

  
  hive.cli.print.current.db
  true
  Whether to include the current database in the Hive 
prompt.
  

  
  hive.metastore.uris
  thrift://hiveserver2http01:9083
  Thrift uri for the remote metastore. Used by metastore 
client to connect to remote metastore.
  

  
  javax.jdo.option.ConnectionDriverName
  com.mysql.jdbc.Driver
  JDBC Driver
  

  
  javax.jdo.option.ConnectionURL
  jdbc:mysql://hivedb01/metastore
  JDBC connect string for a JDBC metastore
  

  
  javax.jdo.option.ConnectionUserName
  metastore
  username to use against metastore database
  

  
  javax.jdo.option.ConnectionPassword
  
  password to use against metastore database
  

  
  datanucleus.autoCreateSchema
  false
  creates necessary schema on a startup if one doesn't exist. 
set this to false, after creating it once
  

  
  hive.metastore.authorization.storage.checks
  true
  Should the metastore do authorization checks against the 
underlying storage
  for operations like drop-partition (disallow the drop-partition if the user in
  question doesn't have permissions to delete the corresponding directory
  on the storage).
  

  
  hive.metastore.sasl.enabled
  true
  If true, the metastore thrift interface will be secured with 
SASL. Clients must authenticate with Kerberos.
  

  
  hive.metastore.kerberos.keytab.file
  /path/to/metastore.keytab
  The path to the Kerberos Keytab file containing the 
metastore thrift server's service principal.
  

  
  hive.metastore.kerberos.principal
  primary/instance@realm
  The service principal for the metastore thrift server. The 
special string _HOST will be replaced automatically with the correct host 
name

[jira] [Created] (HIVE-13818) (Part 2) EOFException with fast hashtable

[Matt McCline (JIRA)]

Matt McCline created HIVE-13818:
---

 Summary: (Part 2) EOFException with fast hashtable
 Key: HIVE-13818
 URL: https://issues.apache.org/jira/browse/HIVE-13818
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


Changes for HIVE-13682 did fix a bug in Fast Hash Tables, but evidently not 
this issue according to Gopal/Rajesh/Nita.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)