Re: Review Request 43921: HIVE-13120 propagate doAs when generating ORC splits

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43921/#review120417
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java (line 545)
<https://reviews.apache.org/r/43921/#comment181851>

not needed


- Sergey Shelukhin


On Feb. 24, 2016, 12:12 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43921/
> ---
> 
> (Updated Feb. 24, 2016, 12:12 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f36f707 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 9b1d7ae 
> 
> Diff: https://reviews.apache.org/r/43921/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 43921: HIVE-13120 propagate doAs when generating ORC splits

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43921/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f36f707 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
9b1d7ae 

Diff: https://reviews.apache.org/r/43921/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
---

(Updated Feb. 26, 2016, 5:04 a.m.)


Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e868500 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
bd4e188 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b5c4d1d 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
cb092d1 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 1b388aa 
  orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 520ae74 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d175d2d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java aa1c925 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
6f84708 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
7a93b54 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
dc71db4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
be5e67b 

Diff: https://reviews.apache.org/r/40415/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
---

(Updated Feb. 26, 2016, 8:12 p.m.)


Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e868500 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java 
ec0be2b 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
bd4e188 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b5c4d1d 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
cb092d1 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 1b388aa 
  orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 520ae74 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d175d2d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java aa1c925 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
6f84708 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
7a93b54 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
dc71db4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
be5e67b 

Diff: https://reviews.apache.org/r/40415/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 44128: HIVE-13002 Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44128/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java f9a9fd2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java a1b98f4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 6fca9f7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 27e7b43 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 2f0a167 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 28bc9a7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java f2b15c5 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook.java 
48f3b28 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionIterable.java b956987 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java 
9f8fb03 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java
 a8ba4d7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 28c8fdb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 0c087ed 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java f323b05 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java fc555ca 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
 dc34507 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java
 a1299a4 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 8e60757 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
 038e5fd 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
4aa1875 

Diff: https://reviews.apache.org/r/44128/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

> On Feb. 16, 2016, 7:38 p.m., Prasanth_J wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java, 
> > line 1523
> > <https://reviews.apache.org/r/40415/diff/3/?file=1240578#file1240578line1523>
> >
> > Then why provide this interface? Is it client's responsibility as to 
> > when to use this interface?

Yes :)


> On Feb. 16, 2016, 7:38 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java, line 484
> > <https://reviews.apache.org/r/40415/diff/3/?file=1240581#file1240581line484>
> >
> > Is it intentional? Stacktrace will be lost.

Yeah, this exception is not very useful, it's usually just the fact that the FS 
doesn't support file IDs.


> On Feb. 16, 2016, 7:38 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java, line 
> > 41
> > <https://reviews.apache.org/r/40415/diff/3/?file=1240583#file1240583line41>
> >
> > what is the byteBuffer arg contain? What is the return type? It's 
> > unclear from the function signature. I think it will be better to do the 
> > conversion from SplitInfo to ByteBuffer just right before we send it across 
> > the wire. SplitInfo can be generic across file formats.

changed the return type; the arg is file metadata, documented on the interface, 
I renamed it for clarity


> On Feb. 16, 2016, 7:38 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java, line 625
> > <https://reviews.apache.org/r/40415/diff/3/?file=1240584#file1240584line625>
> >
> > nit. are there multiple caches?

yes, by conf. Usually there would only be one. Renamed for clarity.


> On Feb. 16, 2016, 7:38 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java, line 1002
> > <https://reviews.apache.org/r/40415/diff/3/?file=1240584#file1240584line1002>
> >
> > same as above. is it required to drop the stacktrace and just print the 
> > message?

Yes, stacktrace is annoying and pointless :) In case if there's some FS issue, 
the other (non-ID) API will probably also fail and produce a proper stack trace


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/#review119116
---


On Feb. 26, 2016, 5:04 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40415/
> ---
> 
> (Updated Feb. 26, 2016, 5:04 a.m.)
> 
> 
> Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e868500 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
> bd4e188 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> b5c4d1d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cb092d1 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
>  1b388aa 
>   orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 520ae74 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java 
> ef76723 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java d175d2d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java aa1c925 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 6f84708 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
> 7a93b54 
>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
> dc71db4 
>   
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
>  be5e67b 
> 
> Diff: https://reviews.apache.org/r/40415/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 42482: HIVE-12855 LLAP UDF whitelist check

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42482/
---

(Updated Jan. 19, 2016, 8:48 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
d6e1a6e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
 bc0ad02 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
c5759d6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 5353062 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 
0e6e2de 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 
6098ddd 

Diff: https://reviews.apache.org/r/42482/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 42482: HIVE-12855 LLAP UDF whitelist check

[Sergey Shelukhin]

> On Jan. 25, 2016, 9:22 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java, 
> > line 78
> > <https://reviews.apache.org/r/42482/diff/3/?file=1202083#file1202083line78>
> >
> > Interesting, so you use this hook to prevent query plans from being 
> > deserialized if they contain UDFs that are not in the whiltelist. Is this 
> > something that should be added to non-LLAP execution, or does this not work 
> > because of temporary functions?

We don't have a similar whitelist for non-LLAP... we can add one and add it, 
but there's no reason to, right now.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42482/#review116173
-------


On Jan. 19, 2016, 8:48 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42482/
> ---
> 
> (Updated Jan. 19, 2016, 8:48 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
> d6e1a6e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
>  bc0ad02 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> c5759d6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 5353062 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 
> 0e6e2de 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 
> 6098ddd 
> 
> Diff: https://reviews.apache.org/r/42482/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

HiveQA machines are low on disk space

[Sergey Shelukhin]

Hi.

Recently, there has occasionally been a stuck mini cluster test, usually
MiniTez, I think I’ve seen MiniSpark too.
Upon looking at the logs for one recent timeout, I don’t see any obvious
problem with Hive, but there’s this:

2016-01-25T09:23:15,658 WARN
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonito
r@6845113[]]: namenode.FSNamesystem (FSNamesystem.java:run(5159)) -
NameNode low on available disk space. Entering safe mode.
2016-01-25T09:23:15,658 INFO
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonito
r@6845113[]]: hdfs.StateChange (FSNamesystem.java:reportStatus(6003)) -
STATE* Safe mode is ON.
Resources are low on NN. Please add or free up more resources then turn
off safe mode manually. NOTE:  If you turn off safe mode before adding
resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin
-safemode leave" to turn safe mode off.



After that, there are many safe-mode related errors.


I think that could be related to recent changes to preserve more logs. Can
we perhaps scale down log preservation a little bit, or gzip them?
It could also be something else on HiveQA machine.

Re: Review Request 42482: HIVE-12855 LLAP UDF whitelist check

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42482/
---

(Updated Jan. 25, 2016, 10:33 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
544af09 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
 bc0ad02 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
7d7fa00 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 5353062 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 
0e6e2de 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 
6098ddd 

Diff: https://reviews.apache.org/r/42482/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Need your opinion on HIVE-12916 to clean up metastore API

[Sergey Shelukhin]

There are 2 separate issues.

1) Removing old APIs. There’s (almost) no impact from keeping the old
function for compat if both old and new call the same internal method with
slight argument variations, but there would be compact problems with
removing them if someone is using it; metastore is used by many projects
across the ecosystem, and I don’t know what functions they might be using.
So I think we should keep the function. So far I’m -1 on removing (or
breaking signature changes).

1a) We can deprecate the functions (in release notes, I presume; Thrift
doesn’t support deprecation as far as I know), then remove them in next
release, but I think we should do it in a systematic manner with entire
metastore API, if there’s will to do the work.

2) Adding new arguments to old APIs.
I’d rather that we add new API based on Request/Response pattern for such
cases, those are easy to evolve. add_partitions vs add_partitions_req is
one example.
Then, as part of 1a, we can also remove the old APIs.
I am -0 on that part of the JIRA, I guess it should be ok to (ab)use
EnvironmentContext for additional parameters ;) As long as the change is
backward compatible.




On 16/1/25, 16:15, "Pengcheng Xiong"  wrote:

>Hi all,
>
>I was working on a new feature in HIVE-12730 and I was suggested by
>Ashutosh to put a configuration in EnvironmentContext. Then I found that
>we
>have lots of cases where the configuration and EnvironmentContext both
>exist in separate functions in the API list. For example, we have
>
>public void alter_table(String dbname, String tbl_name, Table new_tbl)
>throws InvalidOperationException, MetaException,
>org.apache.thrift.TException;
>
>public void alter_table_with_environment_context(String dbname, String
>tbl_name, Table new_tbl, EnvironmentContext environment_context) throws
>InvalidOperationException, MetaException, org.apache.thrift.TException;
>
>public void alter_table_with_cascade(String dbname, String tbl_name, Table
>new_tbl, boolean cascade) throws InvalidOperationException, MetaException,
>org.apache.thrift.TException;
>
> My plan is to keep only
>
>public void alter_table_with_environment_context(String dbname, String
>tbl_name, Table new_tbl, EnvironmentContext environment_context)
>throws InvalidOperationException,
>MetaException, org.apache.thrift.TException;
>
>And then merge the other two into this single one at both client and
>server side. For example, put cascade in the environment_context at client
>side and then read it from server side. If we do so, then it is easy to
>extend in the future, for example,  my current work in HIVE-12730.
>
>   The patch is already there but Sergey has some concerns about backward
>compatibility. Could you please leave your comments/suggestions? Thanks a
>lot.
>
>
>Best
>
>Pengcheng

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41377/
---

(Updated Jan. 21, 2016, 12:27 a.m.)


Review request for hive, Gunther Hagleitner and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 32049eb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java e8864ae 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
3bfe35a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
a2791a1 

Diff: https://reviews.apache.org/r/41377/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 42482: HIVE-12855 LLAP UDF whitelist check

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42482/
---

(Updated Jan. 27, 2016, 1:58 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
544af09 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
 bc0ad02 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
7d7fa00 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 5353062 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java ea9813c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 
0e6e2de 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 
6098ddd 

Diff: https://reviews.apache.org/r/42482/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 43541: HIVE-12856 LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43541/
---

Review request for hive.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ab1dba 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
 2a6ef3a 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
c0d4690 
  metastore/if/hive_metastore.thrift 1abcfe5 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
98fbf70 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f51e1a4 

Diff: https://reviews.apache.org/r/43541/diff/


Testing
---


Thanks,

Sergey Shelukhin

[ANNOUNCE] Apache Hive 2.0.0 Released

[Sergey Shelukhin]

The Apache Hive team is proud to announce the the release of Apache Hive
version 2.0.0.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top of
Apache Hadoop (TM), it provides:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce and Apache Tez frameworks.

For Hive release details and downloads, please visit:
https://hive.apache.org/downloads.html

Hive 2.0.0 Release Notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12332641
jectId=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

branch-2.0 usage past 2.0 release, and release plans

[Sergey Shelukhin]

Hi.
Hive 2.0 having been released, branch-2.0 is now open, but should only
include commits for a hypothetical Hive 2.0.1 bug fix release. As far as I
understand, I will be the RM for all such dot-dot releases for 2.0.
Time-wise, 2.0.1 will be released based on the number and impact of the
bugs found and proposed for inclusion.

master should have the commits intended for the next 2.X feature release,
namely Hive 2.1.
I wonder, given the nature of Hive 2, if we should consider higher cadence
for Hive 2 feature releases now that the first one is out. As usual, each
dot release will have a separate RM, so perhaps we should just allow
people to release when they are willing to take on the work, with some
reasonable minimum time between releases (like a month or two).
Thoughts?

Spark and HBase metastore jiras in 2.0 release

[Sergey Shelukhin]

Hi. 
There is a number of JIRAs resolved in a variety of branches that are a
part of 2.0 release.
We need to close these; I’ll take care of closing the llap JIRAs; as far
as hbase-metastore-branch and spark-branch are concerned, any objection to
closing these? None of them look new. I will close tomorrow if no
objections.

https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20%28
fixVersion%20%3D%20spark-branch%20OR%20fixVersion%20%3D%20hbase-metastore-b
ranch%29%20AND%20fixVersion%20not%20in%20%282.0.0%2C%202.1.0%29%20AND%20sta
tus%20%3D%20Resolved%20ORDER%20BY%20fixVersion%20ASC%2C%20resolved%20DESC

Re: Review Request 43541: HIVE-12856 LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43541/
---

(Updated Feb. 17, 2016, 2:13 a.m.)


Review request for hive.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 48e8491 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
 2a6ef3a 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
e066742 
  metastore/if/hive_metastore.thrift 1abcfe5 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
98fbf70 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 8d05f49 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
61257f0 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f51e1a4 

Diff: https://reviews.apache.org/r/43541/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Spark and HBase metastore jiras in 2.0 release

[Sergey Shelukhin]

Thanks! I have closed all the issues that were updated to 2.0 since
yesterday. Can you bulk close any other issues if adding them to 2.0,
after setting the version?


On 16/2/17, 06:08, "Xuefu Zhang" <xzh...@cloudera.com> wrote:

>I don't think 2.0 branch has every patch from spark-branch. However,
>master
>does. I will take care of spark branch JIrA resolution versions.
>
>Thanks,
>Xuefu
>
>On Tue, Feb 16, 2016 at 9:48 PM, Thejas Nair <thejas.n...@gmail.com>
>wrote:
>
>> The hbase-metastore branch jiras are all part of Hive 2.0.0 release,
>> as no work was done in that branch after the merge into the master
>> branch.
>> I think we should add 2.0.0 as a fix version before closing them.
>> Thoughts ?
>>
>> I am not sure if that is the case with spark-branch. Ideally, I think
>> we should update the fix version whenever we merge the branch into
>> master/branch-1 .
>>
>>
>>
>> On Tue, Feb 16, 2016 at 5:31 PM, Sergey Shelukhin
>> <ser...@hortonworks.com> wrote:
>> > Hi.
>> > There is a number of JIRAs resolved in a variety of branches that are
>>a
>> > part of 2.0 release.
>> > We need to close these; I’ll take care of closing the llap JIRAs; as
>>far
>> > as hbase-metastore-branch and spark-branch are concerned, any
>>objection
>> to
>> > closing these? None of them look new. I will close tomorrow if no
>> > objections.
>> >
>> >
>> 
>>https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20%
>>28
>> >
>> 
>>fixVersion%20%3D%20spark-branch%20OR%20fixVersion%20%3D%20hbase-metastore
>>-b
>> >
>> 
>>ranch%29%20AND%20fixVersion%20not%20in%20%282.0.0%2C%202.1.0%29%20AND%20s
>>ta
>> > 
>>tus%20%3D%20Resolved%20ORDER%20BY%20fixVersion%20ASC%2C%20resolved%20DESC
>> >
>>

Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks some small chunks as "incomplete CB"

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/
---

Review request for hive, Gopal V and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
bae571e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java 
c292b37 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
dbee823 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 eb251a8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
 e970137 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 901e58a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
29b51ec 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 ddba889 

Diff: https://reviews.apache.org/r/45062/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 44491: HIVE-13221 expose metastore APIs from HS2

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44491/
---

(Updated March 9, 2016, 7:29 p.m.)


Review request for hive, Ashutosh Chauhan, Sushanth Sowmyan, and Thejas Nair.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27a56dd 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
0e8a157 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 892a476 

Diff: https://reviews.apache.org/r/44491/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
---

(Updated March 9, 2016, 2:07 a.m.)


Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java 
ec0be2b 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
bd4e188 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
0e8a157 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
9048d45 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
62677d1 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 1b388aa 
  orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
7a93b54 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
dc71db4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
be5e67b 

Diff: https://reviews.apache.org/r/40415/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need collision checks

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
---

(Updated March 9, 2016, 2:32 a.m.)


Review request for hive and Gopal V.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851 
  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 d33f724 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
17d9fdf 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
1132171 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
d2c1907 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 b81e97d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 18191da 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2597848 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 b36cf64 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 2e4e0c5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
 43c8fb3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 8479d22 
  orc/src/java/org/apache/orc/FileMetadata.java d63bdcc 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
6cec80e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java 
da673a5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/StreamUtils.java 9ac53af 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 3ef7abe 

Diff: https://reviews.apache.org/r/44248/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: [ANNOUNCE] New Hive Committer - Wei Zheng

[Sergey Shelukhin]

Congrats!

From: Szehon Ho >
Reply-To: "u...@hive.apache.org" 
>
Date: Wednesday, March 9, 2016 at 17:40
To: "u...@hive.apache.org" 
>
Cc: "dev@hive.apache.org" 
>, 
"w...@apache.org" 
>
Subject: Re: [ANNOUNCE] New Hive Committer - Wei Zheng

Congratulations Wei!

On Wed, Mar 9, 2016 at 5:26 PM, Vikram Dixit K 
> wrote:
The Apache Hive PMC has voted to make Wei Zheng a committer on the Apache Hive 
Project. Please join me in congratulating Wei.

Thanks
Vikram.

Review Request 44491: HIVE-13221 expose metastore APIs from HS2

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44491/
---

Review request for hive, Ashutosh Chauhan, Sushanth Sowmyan, and Thejas Nair.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
50b38fa 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 892a476 

Diff: https://reviews.apache.org/r/44491/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 1222b94 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 1222b94 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45704: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45704/#review126945
---




llap-server/src/main/resources/llap-daemon-log4j2.properties (line 73)
<https://reviews.apache.org/r/45704/#comment190023>

unnecessary



orc/src/java/org/apache/orc/impl/OutStream.java (line 323)
<https://reviews.apache.org/r/45704/#comment190024>

unnecessary


- Sergey Shelukhin


On April 4, 2016, 9:18 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45704/
> ---
> 
> (Updated April 4, 2016, 9:18 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 95c5c0e 
>   data/conf/hive-log4j2.properties 6bace1f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  fb0867d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
>  82187bd 
>   llap-server/src/main/resources/llap-daemon-log4j2.properties 268eb59 
>   orc/src/gen/protobuf-java/org/apache/orc/OrcProto.java 24715c3 
>   orc/src/java/org/apache/orc/OrcConf.java 6fcbb72 
>   orc/src/java/org/apache/orc/OrcFile.java 85506ff 
>   orc/src/java/org/apache/orc/TypeDescription.java bd900ac 
>   orc/src/java/org/apache/orc/impl/BitFieldWriter.java aa5f886 
>   orc/src/java/org/apache/orc/impl/IntegerWriter.java 419054f 
>   orc/src/java/org/apache/orc/impl/OutStream.java 81662cc 
>   orc/src/java/org/apache/orc/impl/RunLengthByteWriter.java 09108b2 
>   orc/src/java/org/apache/orc/impl/RunLengthIntegerWriter.java 3e5f2e2 
>   orc/src/java/org/apache/orc/impl/RunLengthIntegerWriterV2.java fab2801 
>   orc/src/java/org/apache/orc/impl/SerializationUtils.java 2e5a59b 
>   orc/src/java/org/apache/orc/impl/WriterImpl.java f8afe06 
>   orc/src/protobuf/orc_proto.proto f4935b4 
>   orc/src/test/org/apache/orc/impl/TestBitFieldReader.java e4c6f6b 
>   orc/src/test/org/apache/orc/impl/TestBitPack.java f2d3d64 
>   orc/src/test/org/apache/orc/impl/TestInStream.java 9e65345 
>   orc/src/test/org/apache/orc/impl/TestIntegerCompressionReader.java 399f35e 
>   orc/src/test/org/apache/orc/impl/TestOutStream.java e9614d5 
>   orc/src/test/org/apache/orc/impl/TestRunLengthByteReader.java a14bef1 
>   orc/src/test/org/apache/orc/impl/TestRunLengthIntegerReader.java 28239ba 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/PostExecOrcFileDump.java 
> d5d1370 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java 9c2f88f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/JsonFileDump.java 00de545 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java 8a73948 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> f4cfa53 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 85923a8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestJsonFileDump.java acf232d 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestStringDictionary.java 
> 41a211b 
>   ql/src/test/queries/clientpositive/orc_lengths.q PRE-CREATION 
>   ql/src/test/resources/orc-file-dump-bloomfilter.out 18fd2fb 
>   ql/src/test/resources/orc-file-dump-bloomfilter2.out fa5cc2d 
>   ql/src/test/resources/orc-file-dump-dictionary-threshold.out 17a964b 
>   ql/src/test/resources/orc-file-dump.json bf654a1 
>   ql/src/test/resources/orc-file-dump.out 70f7fbd 
>   ql/src/test/resources/orc-file-has-null.out e98a73f 
>   ql/src/test/results/clientpositive/acid_globallimit.q.out d43ed42 
>   ql/src/test/results/clientpositive/alter_merge_orc.q.out b5a6d04 
>   ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out 0d5ba01 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 131cf6a 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out 6db4ded 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> 179bc66 
>   ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
> d03bfe4 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 3b24a2e 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out 
> a30c356 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 
> 4e589b8 
>   ql/

Re: [DISCUSS] Hive 2.0.1 bug fix release

[Sergey Shelukhin]

The vote has passed; I’ve marked some bugs fixed for 2.1 to target 2.0.1
and backported a few patches.
Please mark any bug fixes in progress that you think should be in 2.0.1,
for 2.0.1.

On 16/3/31, 22:58, "Carl Steinbach" <cwsteinb...@gmail.com> wrote:

>+1
>
>On Thu, Mar 31, 2016 at 9:06 PM, Prasanth J <j.prasant...@gmail.com>
>wrote:
>
>> +1
>>
>> Thanks
>> Prasanth
>> > On Mar 31, 2016, at 5:37 PM, Lefty Leverenz <leftylever...@gmail.com>
>> wrote:
>> >
>> > +1
>> >
>> > -- Lefty
>> >
>> > On Thu, Mar 31, 2016 at 4:55 PM, Thejas Nair <thejas.n...@gmail.com>
>> wrote:
>> >
>> >> +1
>> >>
>> >> On Thu, Mar 31, 2016 at 1:20 PM, Sergey Shelukhin
>> >> <ser...@hortonworks.com> wrote:
>> >>> There have been a few bug fixes since 2.0, some more might be in the
>> >>> pipeline.
>> >>> I think it’s a good time to release 2.0.1.
>> >>> I will go thru the log diff between master and branch-2 and see if
>> >>> anything needs to be backported, people can also mark JIRAs as
>> >>> 2.0.1-targeted.
>> >>> Depending on how many of the latter we get, I’d expect to cut an RC
>>in
>> >> 1-3
>> >>> weeks (hopefully this one will be much simpler than 2.0).
>> >>>
>> >>> Any objections/comments? If not, I’ll start a release plan vote
>>thread
>> >>> tomorrow.
>> >>>
>> >>
>>
>>

recent metastore failures in HiveQA

[Sergey Shelukhin]

Has anyone else noticed that many tests that involve metastore started
failing lately? The failures are sporadic and happen both in the tests
that test metastore, and q files that use metastore…
The error is always something like
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
ava:198)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
Utils.java:1208)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
ls.java:1195)
at 
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
etrics.java:54)

I wonder if someone has insight on whether this is an environment issue,
or someone broke something recently, before we investigate more :)

Re: Review Request 45704: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 1222b94 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45704/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

> On April 6, 2016, 11:27 p.m., Prasanth_J wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 1205-1206
> > <https://reviews.apache.org/r/45238/diff/1/?file=1312350#file1312350line1205>
> >
> > IMO, this should not be made an option for the user. Should be enabled 
> > as long as index is enabled. If for some reason, this is wrong, then disk 
> > range computation should fallback to old code path using WORST_CASE_SLOP (2 
> > x buffer size). Will be better to have config for fallback path as opposed 
> > to tracking lengths.

this is needed for the write path - what if the write path is problematic 
because of the length tracking?


> On April 6, 2016, 11:27 p.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/TypeDescription.java, line 384
> > <https://reviews.apache.org/r/45238/diff/1/?file=1312356#file1312356line384>
> >
> > Is this related?

yes


> On April 6, 2016, 11:27 p.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/RunLengthIntegerWriterV2.java, line 192
> > <https://reviews.apache.org/r/45238/diff/1/?file=1312362#file1312362line192>
> >
> > By flush does this mean writing to OutStream? What if numLiterals 
> > straddles compression buffers (first CB is flushed and closed and second CB 
> > contains some literals)? Is that still considered to be flushed?

yes, just writing to outstream. It doesn't matter for this class what happens 
there for the most part (see big comment in WriterImpl); the flush count is 
always updated after writing out to OutStream, so in any callback from CBs 
created during the write, we'll still see the old flush count. Only when that 
2nd CB is written, we will get a callback and find the new flush count.


> On April 6, 2016, 11:27 p.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/WriterImpl.java, line 595
> > <https://reviews.apache.org/r/45238/diff/1/?file=1312364#file1312364line595>
> >
> > I think it will be easier if we maintain a Map<String, LengthsBuilder> 
> > that maps stream name and proto builder of lengths. The callback interface 
> > can be reportCompressionBuffer(String name, int cbSize). When this callback 
> > is called, length information can be appended to the lengths builder for 
> > the corresponding stream. When flushing the stripe, using the same stream 
> > name we can check if the stream is suppressed or not. With that we can also 
> > avoid the index adjustment. 
> > 
> > I am just seeing if there are ways to simplify the state maintenance 
> > here. Thoughts?

the map by enum might make sense... do you think this is acceptable in terms of 
extra overhead?


> On April 6, 2016, 11:27 p.m., Prasanth_J wrote:
> > orc/src/protobuf/orc_proto.proto, line 85
> > <https://reviews.apache.org/r/45238/diff/1/?file=1312365#file1312365line85>
> >
> > If this is a required field, it will break backward compatibility. 
> > Isn't it?

it's repeated, not required :)


> On April 6, 2016, 11:27 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java, line 
> > 280
> > <https://reviews.apache.org/r/45238/diff/1/?file=1312378#file1312378line280>
> >
> > I don't see any changes that bypasses this when lengths are available. 
> > Is that going into separate jira?

see estimateRgEndOffset calls. It calls the fn right now to log the message, 
but uses lengths if available. I can remove it when the patch is done... or 
only do it if info is enabled


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45238/#review126974
---


On March 23, 2016, 7:08 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45238/
> ---
> 
> (Updated March 23, 2016, 7:08 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c14df20 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java
>  c292b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb251a8 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
>  82187bd 
>   orc/src/ja

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 1222b94 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Reviews & commits (RTC/CTR), contributions, bylaws

[Sergey Shelukhin]

2 days may be too short… One can submit all the patches on Friday night
and commit on Sunday ;)

On 16/4/11, 17:43, "Prasanth Jayachandran" 
wrote:

>Also, it will be good to add “NO PRECOMMIT TESTS” in the description of
>the jira
>if the patches are really small and does not affect any tests
>(checkstyles, typos etc.)
>to avoid occupying a slot in the precommit queue which could be
>overloaded sometimes.
>
>+1 for 2 days no response on dev-list.
>
>Thanks
>Prasanth
>
>> On Apr 11, 2016, at 7:36 PM, Lefty Leverenz 
>>wrote:
>> 
>>> 
>>> Maybe the process can be to solicit reviews for such minor patches by
>>> sending an email to dev@ list and if no response is seen in 2 days, go
>>> ahead and commit it ?
>>> 
>> 
>> Two days seems reasonable, perhaps excluding weekends and major
>>holidays.
>> 
>> -- Lefty
>> 
>> On Mon, Apr 11, 2016 at 1:26 PM, Thejas Nair 
>>wrote:
>> 
>>> I agree we have a problem here. At least patches as small as this
>>> shouldn't take too long to get reviewed.
>>> 
>>> Knox seems to consider a very large set of patches as being under CTR
>>> process.
>>> I think hive is very large and mature project that I would lean
>>> towards RTC process for most issues. I think we can make an exception
>>> for very minor patches such as fixing typos and and checkstyle issues.
>>> Maybe the process can be to solicit reviews for such minor patches by
>>> sending an email to dev@ list and if no response is seen in 2 days, go
>>> ahead and commit it ?
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Apr 11, 2016 at 6:38 AM, Lars Francke 
>>> wrote:
 Hi,
 
 I've been a long-time contributor to Hive (5 or so years) and have
been
 voted in as a committer and I'm very grateful for that. I also
understand
 that my situation is different than most or lots of committers as I'm
not
 working for one of the big companies (Facebook, Cloudera, Hortonworks
>>> etc.)
 where you can just ask someone sitting next to you to do a review.
 
 I'd really like to contribute more than I do currently but the
process of
 getting patches in is painful for me (and other 'outside'
contributors)
>>> as
 it is hard to get reviews & things committed. The nature of most of my
 patches is very minor[1] (fixing typos, checkstyle issues etc.) and I
 understand that these are not the most interesting patches to review
and
 are easy to miss. I don't blame anyone for this situation as I totally
 understand it and have been on the other side of this for other
projects.
 
 Is there anything we can do to make it easier for me and others like
me
>>> to
 contribute here? I absolutely see the value in having "cleaner" code
and
 when done in small batches it's usually not very disruptive either.
 
 The bylaws currently require a +1 from a committer who has not
authored
>>> the
 patch. Knox for example has a different policy [2] where they
distinguish
 between major features and minor things which can be committed freely.
 
 Hive could adopt something similar or like a middle ground. These are
>>> just
 two suggestions:
 
 1) Allow minor changes (up to the committers discretion) without
>>> requiring
 an extra +1
 2) Allow minor changes (up to the committers discretion) with Lazy
>>> approval
 (i.e. wait 24 hours)
 
 Sorry for the long rant but I'd love some feedback on this and am
looking
 forward to contributing more in the future.
 
 Cheers,
 Lars
 
 [1] e.g. 
 [2] <
>>> https://cwiki.apache.org/confluence/display/KNOX/Contribution+Process>
>>> 
>

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out f4e21bd 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 1222b94 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/spark/bucket5.q.out 5baf054 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out 
a343d93 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out c70f104 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out f4e21bd 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 46048: HIVE-13194 remove sessionHive

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46048/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java abf94ff 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java f6fd081 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c27481f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/TableIterable.java f3af39b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 19342a8 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java f7a0f31 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 16d7d3f 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
5b8ec60 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java
 d795324 
  service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
  
service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java 
35b6c52 
  
service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
 8db2e62 
  
service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java 
d6f6280 
  
service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java 
740b851 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
9ce6055 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
80a1844 
  
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
 025b0b8 

Diff: https://reviews.apache.org/r/46048/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46048: HIVE-13194 remove sessionHive

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46048/
---

(Updated April 11, 2016, 8:51 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java abf94ff 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c27481f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/TableIterable.java f3af39b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 19342a8 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java
 d795324 
  
service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java 
35b6c52 
  
service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
 8db2e62 
  
service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java 
d6f6280 
  
service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java 
740b851 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
9ce6055 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
80a1844 

Diff: https://reviews.apache.org/r/46048/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review124234
---




jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 210)
<https://reviews.apache.org/r/40867/#comment186691>

nit: whitespace



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1917)
<https://reviews.apache.org/r/40867/#comment186693>

why cannot the conf itself be passed in? why is the map necessary?
Or, if that doesn't work for some reason, what about creating a small 
struct class to hold the fields with logical names and types, instead of the 
map.


- Sergey Shelukhin


On March 16, 2016, 8:51 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated March 16, 2016, 8:51 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d9a273b 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 04d816a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 8baecdf 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 62fcde5 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks some small chunks as "incomplete CB"

[Sergey Shelukhin]

> On March 22, 2016, 5:09 a.m., Gopal V wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java,
> >  line 844
> > <https://reviews.apache.org/r/45062/diff/1/?file=1307057#file1307057line844>
> >
> > is Data ever non-null here?
> 
> Sergey Shelukhin wrote:
> Yes, in EncodedReaderImpl
>  // 2.5. Remember the bad estimates for future reference.
> if (badEstimates != null && !badEstimates.isEmpty()) {
>   // Relies on the fact that cache does not actually store these.
>   DiskRange[] cacheKeys = badEstimates.toArray(new 
> DiskRange[badEstimates.size()]);
>   long[] result = cacheWrapper.putFileData(fileKey, cacheKeys, null, 
> baseOffset);
>   assert result == null; // We don't expect conflicts from bad 
> estimates.
> }

and for non-null:
// 6. Finally, put uncompressed data to cache.
if (fileKey != null) {
  long[] collisionMask = cacheWrapper.putFileData(fileKey, cacheKeys, 
targetBuffers, baseOffset);
  processCacheCollisions(collisionMask, toDecompress, targetBuffers, 
csd.getCacheBuffers());
}


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/#review124732
-------


On March 18, 2016, 11:18 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45062/
> ---
> 
> (Updated March 18, 2016, 11:18 p.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  bae571e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java
>  c292b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> dbee823 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb251a8 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
>  e970137 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  901e58a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 29b51ec 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  ddba889 
> 
> Diff: https://reviews.apache.org/r/45062/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks some small chunks as "incomplete CB"

[Sergey Shelukhin]

> On March 22, 2016, 5:09 a.m., Gopal V wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java,
> >  line 844
> > <https://reviews.apache.org/r/45062/diff/1/?file=1307057#file1307057line844>
> >
> > is Data ever non-null here?

Yes, in EncodedReaderImpl
 // 2.5. Remember the bad estimates for future reference.
if (badEstimates != null && !badEstimates.isEmpty()) {
  // Relies on the fact that cache does not actually store these.
  DiskRange[] cacheKeys = badEstimates.toArray(new 
DiskRange[badEstimates.size()]);
  long[] result = cacheWrapper.putFileData(fileKey, cacheKeys, null, 
baseOffset);
  assert result == null; // We don't expect conflicts from bad estimates.
}


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/#review124732
-------


On March 18, 2016, 11:18 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45062/
> ---
> 
> (Updated March 18, 2016, 11:18 p.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  bae571e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java
>  c292b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> dbee823 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb251a8 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
>  e970137 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  901e58a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 29b51ec 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  ddba889 
> 
> Diff: https://reviews.apache.org/r/45062/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45238/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c14df20 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java 
c292b37 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 eb251a8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 82187bd 
  orc/src/java/org/apache/orc/OrcConf.java 6fcbb72 
  orc/src/java/org/apache/orc/OrcFile.java 3945a5d 
  orc/src/java/org/apache/orc/TypeDescription.java bd900ac 
  orc/src/java/org/apache/orc/impl/BitFieldWriter.java aa5f886 
  orc/src/java/org/apache/orc/impl/IntegerWriter.java 419054f 
  orc/src/java/org/apache/orc/impl/OutStream.java 81662cc 
  orc/src/java/org/apache/orc/impl/RunLengthByteWriter.java 09108b2 
  orc/src/java/org/apache/orc/impl/RunLengthIntegerWriter.java 3e5f2e2 
  orc/src/java/org/apache/orc/impl/RunLengthIntegerWriterV2.java fab2801 
  orc/src/java/org/apache/orc/impl/SerializationUtils.java c1162e4 
  orc/src/java/org/apache/orc/impl/WriterImpl.java 6497ecf 
  orc/src/protobuf/orc_proto.proto f4935b4 
  orc/src/test/org/apache/orc/impl/TestBitFieldReader.java e4c6f6b 
  orc/src/test/org/apache/orc/impl/TestBitPack.java f2d3d64 
  orc/src/test/org/apache/orc/impl/TestInStream.java 9e65345 
  orc/src/test/org/apache/orc/impl/TestIntegerCompressionReader.java 399f35e 
  orc/src/test/org/apache/orc/impl/TestOutStream.java e9614d5 
  orc/src/test/org/apache/orc/impl/TestRunLengthByteReader.java a14bef1 
  orc/src/test/org/apache/orc/impl/TestRunLengthIntegerReader.java 28239ba 
  ql/src/java/org/apache/hadoop/hive/llap/DebugUtils.java ea626d7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 67f9da8 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PostExecOrcFileDump.java d5d1370 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java 9c2f88f 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/JsonFileDump.java 00de545 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java 8a73948 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
96af96a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
29b51ec 
  ql/src/test/queries/clientpositive/orc_lengths.q PRE-CREATION 
  ql/src/test/results/clientpositive/orc_lengths.q.out PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 ddba889 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need collision checks

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
---

(Updated March 3, 2016, 11:20 p.m.)


Review request for hive and Gopal V.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad 
  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 d33f724 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
17d9fdf 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
1132171 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
d2c1907 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 b81e97d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 18191da 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2597848 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 b36cf64 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 2e4e0c5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
 43c8fb3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 8479d22 
  orc/src/java/org/apache/orc/FileMetadata.java d63bdcc 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
6cec80e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java 
da673a5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 3ef7abe 

Diff: https://reviews.apache.org/r/44248/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need collision checks

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
---

Review request for hive and Gopal V.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
17d9fdf 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
1132171 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
d2c1907 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 b81e97d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 18191da 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2597848 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 b36cf64 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 2e4e0c5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
 43c8fb3 
  orc/src/java/org/apache/orc/FileMetadata.java d63bdcc 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
6cec80e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java 
da673a5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 3ef7abe 

Diff: https://reviews.apache.org/r/44248/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
---

(Updated March 2, 2016, 2:30 a.m.)


Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java 
ec0be2b 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
bd4e188 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b5c4d1d 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
cb092d1 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 1b388aa 
  orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
7a93b54 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
dc71db4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
be5e67b 

Diff: https://reviews.apache.org/r/40415/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
---

(Updated March 7, 2016, 9:32 p.m.)


Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java 
ec0be2b 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
bd4e188 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b5c4d1d 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
cb092d1 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 1b388aa 
  orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ad17096 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
7a93b54 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
dc71db4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
be5e67b 

Diff: https://reviews.apache.org/r/40415/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
---

(Updated March 7, 2016, 9:36 p.m.)


Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java 
ec0be2b 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
bd4e188 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b5c4d1d 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
cb092d1 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 1b388aa 
  orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ad17096 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
7a93b54 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
dc71db4 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
be5e67b 

Diff: https://reviews.apache.org/r/40415/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40415: HIVE-11675 make use of file footer PPD API in ETL strategy or separate strategy

[Sergey Shelukhin]

> On March 7, 2016, 7:27 a.m., Prasanth_J wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java,
> >  line 2223
> > <https://reviews.apache.org/r/40415/diff/6/?file=1276199#file1276199line2223>
> >
> > use ThreadLocal?

Not sure that helps. Also, it's a threadlocal... Remembering the result for 
some time could be a good solution in a follow-up, but we assume by default 
that if the user enables it they know what they are doing, or pay a small perf 
penalty


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/#review122261
---


On March 2, 2016, 2:30 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40415/
> ---
> 
> (Updated March 2, 2016, 2:30 a.m.)
> 
> 
> Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad 
>   metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java 
> ec0be2b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
> bd4e188 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> b5c4d1d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cb092d1 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
>  1b388aa 
>   orc/src/java/org/apache/orc/impl/InStream.java b1c6de5 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java 
> ef76723 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 4fafe8c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 
> 7a93b54 
>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
> dc71db4 
>   
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
>  be5e67b 
> 
> Diff: https://reviews.apache.org/r/40415/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[DISCUSS] Hive 2.0.1 bug fix release

[Sergey Shelukhin]

There have been a few bug fixes since 2.0, some more might be in the
pipeline.
I think it’s a good time to release 2.0.1.
I will go thru the log diff between master and branch-2 and see if
anything needs to be backported, people can also mark JIRAs as
2.0.1-targeted.
Depending on how many of the latter we get, I’d expect to cut an RC in 1-3
weeks (hopefully this one will be much simpler than 2.0).

Any objections/comments? If not, I’ll start a release plan vote thread
tomorrow.

Review Request 45704: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/clientpositive/spark/infer_bucket_sort_map_operators.q.out 
a343d93 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 0eb9132 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 1222b94 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45704/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46597/
---

Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4ab7b32 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 23980d0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 2e99a28 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java 
fc29371 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 e80fb15 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
a0250cb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 e99e689 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
64c2b58 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
14657e6 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/Scheduler.java 
26c8e55 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 57dd828 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 4d05c35 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 a65bf5c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 3ea5ef9 

Diff: https://reviews.apache.org/r/46597/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/
---

(Updated April 23, 2016, 1:57 a.m.)


Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 926806b 
  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 e99e689 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 

Diff: https://reviews.apache.org/r/46579/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46597/
---

(Updated April 23, 2016, 2:08 a.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4ab7b32 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 23980d0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 2e99a28 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java 
fc29371 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 e80fb15 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
a0250cb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 e99e689 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
64c2b58 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
14657e6 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/Scheduler.java 
26c8e55 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 57dd828 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
610f266 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 4d05c35 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 a65bf5c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 3ea5ef9 

Diff: https://reviews.apache.org/r/46597/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

> On April 23, 2016, 12:38 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java,
> >  line 193
> > <https://reviews.apache.org/r/46579/diff/1/?file=1357677#file1357677line193>
> >
> > Can we restrict this configuration parameter to be used within HS2 only 
> > ?
> > Any other client (CLI) would have it off by default - and fetch the 
> > token from LLAP itself.
> 
> Siddharth Seth wrote:
> Also. the config property name will likely need a rename if this is 
> restricted to HS2.

changed to have both options


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/#review130230
-------


On April 22, 2016, 6:47 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46579/
> ---
> 
> (Updated April 22, 2016, 6:47 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
> Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fabb8ab 
>   llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  e99e689 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  76ba225 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 8c7a539 
> 
> Diff: https://reviews.apache.org/r/46579/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

[Sergey Shelukhin]

> On April 25, 2016, 9:50 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java,
> >  line 32
> > <https://reviews.apache.org/r/46597/diff/2/?file=1358356#file1358356line32>
> >
> > Throw something other than IOException, which typically indicates a 
> > network communicationn error in the Hadoop world.

IOException is thrown from UGI, we throw SecurityException now. IOException in 
Hadoop world typically indicates any error. E.g. Hadoop AccessControlException 
is IOException, and that's just starting with "A" ;)


> On April 25, 2016, 9:50 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java,
> >  line 286
> > <https://reviews.apache.org/r/46597/diff/2/?file=1358357#file1358357line286>
> >
> > Could you please add a TODO/jira to get rid of this, and get this 
> > either from the request itself, or from QueryTracker. The scheduler should 
> > not be accesssed to get information like this.

You cannot get this from a request, because what you are verifying is the one 
from a request... QueryTracker would presumably require some refactoring. I 
will add a TODO


> On April 25, 2016, 9:50 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java,
> >  line 151
> > <https://reviews.apache.org/r/46597/diff/2/?file=1358358#file1358358line151>
> >
> > MiniLlap can set this up explicitly. When running outside of YARN - 
> > this will end up having to be setup as an environment variable.
> > 
> > Instead of going the UUID route, I think we should set this up in 
> > MiniLLAP, and fail under regular use - instead of generating a random 
> > string which could lead to all kinds of crazy behaviour in a multi-node 
> > setup.

I changed it to null in both non-YARN paths


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46597/#review130369
---


On April 23, 2016, 2:08 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46597/
> ---
> 
> (Updated April 23, 2016, 2:08 a.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  4ab7b32 
>   llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  23980d0 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  2e99a28 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java 
> fc29371 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  e80fb15 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> a0250cb 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  e99e689 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> 64c2b58 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  14657e6 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/Scheduler.java 
> 26c8e55 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  57dd828 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  76ba225 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 8c7a539 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 610f266 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  4d05c35 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  a65bf5c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 3ea5ef9 
> 
> Diff: https://reviews.apache.org/r/46597/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46597/
---

(Updated April 26, 2016, 12:58 a.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4ab7b32 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 23980d0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 2e99a28 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java 
fc29371 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 e80fb15 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
33b41e8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 e99e689 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
64c2b58 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
14657e6 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/Scheduler.java 
26c8e55 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 57dd828 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
610f266 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 4d05c35 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 a65bf5c 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 3ea5ef9 

Diff: https://reviews.apache.org/r/46597/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/results/clientpositive/spark/bucket5.q.out 0504eac 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out 
a343d93 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
3040544 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 3f5f649 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 7b56585 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out c70f104 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out f4e21bd 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 944a494 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 1365626 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out fc4351d 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 46715: HIVE-13596 HS2 should be able to get UDFs on demand from metastore

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46715/
---

Review request for hive and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bae3999 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java d5f4a37 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 672df63 

Diff: https://reviews.apache.org/r/46715/diff/


Testing
---


Thanks,

Sergey Shelukhin

please withhold commits to Hive branch-2.0

[Sergey Shelukhin]

I’ve just cut the RC0 for Hive 2.0.1 release. Please withhold commits to
Hive branch-2.0 until the release is completed, or the RC is sank.

[VOTE] Apache Hive 2.0.1 Release Candidate 0

[Sergey Shelukhin]

Apache Hive 2.0.1 Release Candidate 0 is available here:

http://home.apache.org/~sershe/hive-2.0.1-rc0/


Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1048/


Source tag for RC0 (github mirror) is:
https://github.com/apache/hive/releases/tag/release-2.0.1-rc0


Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: [VOTE] Apache Hive 2.0.1 Release Candidate 0

[Sergey Shelukhin]

RC is invalid. I will publish RC1 in due course...

On 16/4/27, 15:05, "Sergey Shelukhin" <ser...@hortonworks.com> wrote:

>Apache Hive 2.0.1 Release Candidate 0 is available here:
>
>http://home.apache.org/~sershe/hive-2.0.1-rc0/
>
>
>Maven artifacts are available here:
>
>https://repository.apache.org/content/repositories/orgapachehive-1048/
>
>
>Source tag for RC0 (github mirror) is:
>https://github.com/apache/hive/releases/tag/release-2.0.1-rc0
>
>
>Voting will conclude in 72 hours.
>
>Hive PMC Members: Please test and vote.
>
>Thanks.
>
>
>
>

Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/
---

Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5360ed4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
 73f94f3 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 6af30d4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 e80fb15 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
33b41e8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 e99e689 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 2a60123 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
6a72b4c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 b3b571d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 24f4442 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 08ee769 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
0584ad8 

Diff: https://reviews.apache.org/r/46754/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/#review130862
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java (line 61)
<https://reviews.apache.org/r/46754/#comment194768>

unneeded


- Sergey Shelukhin


On April 27, 2016, 11:01 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46754/
> ---
> 
> (Updated April 27, 2016, 11:01 p.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5360ed4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
>  73f94f3 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  6af30d4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  e80fb15 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 33b41e8 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  e99e689 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  2a60123 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 6a72b4c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  b3b571d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  76ba225 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 8c7a539 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  24f4442 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  08ee769 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 0584ad8 
> 
> Diff: https://reviews.apache.org/r/46754/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46715: HIVE-13596 HS2 should be able to get UDFs on demand from metastore

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46715/
---

(Updated April 28, 2016, 12:20 a.m.)


Review request for hive and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 49d748c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java d5f4a37 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 672df63 

Diff: https://reviews.apache.org/r/46715/diff/


Testing
---


Thanks,

Sergey Shelukhin

new HiveQA jenkins queue doesn't have JIRA numbers

[Sergey Shelukhin]

Subj. Is it a limitation of the new Jenkins instance, or can it be enabled
again? The JIRA number associated with the run used to be shown for the
pending items in the left column.

Review Request 46809: HIVE-13442 LLAP: refactor submit API to be amenable to signing

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46809/
---

Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs
-

  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java ec6e439 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
d4cdac1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 3d45c7a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryFragmentInfo.java
 480a394 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
64c2b58 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
14657e6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 f621af2 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 fcfa940 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 24f4442 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 08ee769 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 a4f5d4d 

Diff: https://reviews.apache.org/r/46809/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/
---

Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fabb8ab 
  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 e99e689 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 

Diff: https://reviews.apache.org/r/46579/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

> On April 12, 2016, 12:52 a.m., Prasanth_J wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1205
> > <https://reviews.apache.org/r/45238/diff/7/?file=1339688#file1339688line1205>
> >
> > Can you make the prefix consistent? rename
> > hive.exec.orc.disk.range.use.lengths? As disk range computation is the 
> > only place where this lengths are used.

added exec; I don't think adding disk range is a good idea; that's the only 
place where you could use them in the first place. Also if someone does come up 
with different usage it should be the same variable.


> On April 12, 2016, 12:52 a.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/BitFieldWriter.java, line 44
> > <https://reviews.apache.org/r/45238/diff/7/?file=1339696#file1339696line44>
> >
> > I don't understand 2 things here.
> > 1) flushedValuesDuringWriter is always set to -1 in the second 
> > assignment
> > 2) bitsLeft will always be 0. See below writeByte is called when 
> > bitsLeft == 0. In another place bitsLeft is set to 0 before calling 
> > writeByte
> > 
> > This seems to be redundant. Is that correct?

(1) is correct. We just save it when the write call is executing, if someone 
wants to find out the number of values then; the normal formula may break 
during write.
(2) is not; flush calls it without bitsLeft being equal to 0


> On April 12, 2016, 12:52 a.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/BitFieldWriter.java, line 66
> > <https://reviews.apache.org/r/45238/diff/7/?file=1339696#file1339696line66>
> >
> > this is not needed as writeByte() sets bitsLeft=8 anyways.

needed for writeBytes to update the value correctly (see the response above)


> On April 12, 2016, 12:52 a.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/RunLengthByteWriter.java, line 67
> > <https://reviews.apache.org/r/45238/diff/7/?file=1339699#file1339699line67>
> >
> > Can this be moved outside the condition? Every write() call is going to 
> > increment this, so better to update this value once outside the condition.

That is not quite true. Added a comment.


> On April 12, 2016, 12:52 a.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/WriterImpl.java, line 668
> > <https://reviews.apache.org/r/45238/diff/7/?file=1339703#file1339703line668>
> >
> > Should this come at the top? or even before this method invocation? I 
> > guess the removeIsPresentPositions will be called regardless if isTracking 
> > is enabled or disabled. Is that correct?

Yes


> On April 12, 2016, 12:52 a.m., Prasanth_J wrote:
> > orc/src/java/org/apache/orc/impl/WriterImpl.java, line 911
> > <https://reviews.apache.org/r/45238/diff/7/?file=1339703#file1339703line911>
> >
> > Should we iterate only if isTrackingLengths is true?

Yes, because the trackers can also be used for the dictionary. Added a comment.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45238/#review128268
---


On April 11, 2016, 7:16 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45238/
> ---
> 
> (Updated April 11, 2016, 7:16 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 95c5c0e 
>   data/conf/hive-log4j2.properties 6bace1f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  fb0867d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
>  82187bd 
>   orc/src/gen/protobuf-java/org/apache/orc/OrcProto.java 24715c3 
>   orc/src/java/org/apache/orc/OrcConf.java 6fcbb72 
>   orc/src/java/org/apache/orc/OrcFile.java 85506ff 
>   orc/src/java/org/apache/orc/TypeDescription.java bd900ac 
>   orc/src/java/org/apache/orc/impl/BitFieldWriter.java aa5f886 
>   orc/src/java/org/apache/orc/impl/IntegerWriter.java 419054f 
>   orc/src/java/org/apache/orc/impl/OutStream.java 81662cc 
>   orc/src/java/org/apache/orc/impl/RunLengthByteWriter.java 09108b2 
>   orc/src/java/org/apache/orc/impl/RunLengthIntegerWriter.java 3e5f2e2 
>   orc/src/java/org/apache/orc/impl/RunLengthIntegerWriterV2.java fab2801 
>   orc/src/

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review130014
---




jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 557)
<https://reviews.apache.org/r/40867/#comment193626>

nit: is it enough to just use Arrays.asList?



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 580)
<https://reviews.apache.org/r/40867/#comment193630>

why does this swallow the error? It should fail on error



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 581)
<https://reviews.apache.org/r/40867/#comment193629>

nit: null check, there could be an error before stream is created


- Sergey Shelukhin


On April 15, 2016, 11:38 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated April 15, 2016, 11:38 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 857805a 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> b7d6549 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 9ce6055 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 0f36cd6 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> be9833d 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 45238: HIVE-9660 store end offset of compressed data for RG in RowIndex in ORC

[Sergey Shelukhin]

/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/spark/bucket5.q.out 5baf054 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out 
a343d93 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out
 d30c1f0 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out 
6b3c375 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out b7f1a65 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out 86df0a7 
  ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out 83988d3 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f0a 
  ql/src/test/results/clientpositive/tez/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/tez/alter_merge_stats_orc.q.out 0d5ba01 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f6 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
97f59d9 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out c70f104 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out f4e21bd 
  ql/src/test/results/clientpositive/tez/llap_nullscan.q.out 39f04ea 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out 87855fa 
  ql/src/test/results/clientpositive/tez/orc_merge10.q.out bcba1bd 
  ql/src/test/results/clientpositive/tez/orc_merge11.q.out 8a4d8e9 
  ql/src/test/results/clientpositive/tez/orc_merge12.q.out f23be5a 
  ql/src/test/results/clientpositive/tez/schema_evol_stats.q.out d396a61 
  ql/src/test/results/clientpositive/tez/union_fast_stats.q.out 578205e 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 3d1f22f 
  ql/src/test/results/clientpositive/union_fast_stats.q.out f0879af 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 3b17591 

Diff: https://reviews.apache.org/r/45238/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/
---

(Updated May 20, 2016, 1:49 a.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9cc8fbe 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 cffa493 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
de817e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
fea3dc7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 b3b571d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java
 PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 279baf1 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 

Diff: https://reviews.apache.org/r/46754/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java,
> >  line 267
> > <https://reviews.apache.org/r/46956/diff/2/?file=1387296#file1387296line267>
> >
> > A follow up jira may be to limit the age of keys.
> > i.e. if a keyId is older than a certain amount of time - fail the 
> > request. I'm not sure how ZKSecretManager rotates these keys, and when they 
> > are invalidated.
> > 
> > A user can potentially use an old (presumably compromsied key) to 
> > generate requests - which will be valid if keys are not rotated/aged.

https://issues.apache.org/jira/browse/HIVE-13820


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
-------


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 47735: HIVE-13617 LLAP: support non-vectorized execution in IO

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47735/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 298f788 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java afe5ee2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 636f079 
  ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/BatchToRowReader.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cfedf35 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 33fe3b6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOiBatchToRowReader.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java f034812 
  ql/src/test/queries/clientpositive/orc_llap_nonvector.q PRE-CREATION 
  ql/src/test/results/clientpositive/orc_llap_nonvector.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/47735/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 290
> > <https://reviews.apache.org/r/46956/diff/2/?file=1387298#file1387298line290>
> >
> > What user is expected over here.
> > 1. In case of an invocation by HS2 to run a Tez query - I'm assuming 
> > this would be the HS2 service user (which is the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.
> 
> Sergey Shelukhin wrote:
> This is the calling user in case of RPC.
> 
> Siddharth Seth wrote:
> This goes back to whether it's invoked locally or remotely.
> Local and Remote calls by HS2 to obtain a token on behalf of an external 
> client, will need to pass in the user name to generate the token correctly. 
> What's obtained on the RPC call will almost always be the Hive user - at 
> least for the remote call.
> Behaviour should not change if the flag to get the token locally / 
> remotely is changed.

This is protobuf, it's always invoked remotely. Two paths only meet at 
SecretManager level.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/
---

(Updated May 23, 2016, 10:22 p.m.)


Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c0843b9 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
 ebc91b1 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 7c47f0b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
5ab7b3c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 04df929 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
c55436b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 e0f0676 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 

Diff: https://reviews.apache.org/r/46956/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java,
> >  line 290
> > <https://reviews.apache.org/r/46956/diff/2/?file=1387298#file1387298line290>
> >
> > What user is expected over here.
> > 1. In case of an invocation by HS2 to run a Tez query - I'm assuming 
> > this would be the HS2 service user (which is the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.
> 
> Sergey Shelukhin wrote:
> This is the calling user in case of RPC.
> 
> Siddharth Seth wrote:
> This goes back to whether it's invoked locally or remotely.
> Local and Remote calls by HS2 to obtain a token on behalf of an external 
> client, will need to pass in the user name to generate the token correctly. 
> What's obtained on the RPC call will almost always be the Hive user - at 
> least for the remote call.
> Behaviour should not change if the flag to get the token locally / 
> remotely is changed.
> 
> Sergey Shelukhin wrote:
> This is protobuf, it's always invoked remotely. Two paths only meet at 
> SecretManager level.
> 
> Siddharth Seth wrote:
> Reference was to the way the token is obtained (locally or remotely) - 
> not the specific call.

fixed by only retaining the remote API for CLI and removing the setting. HS2 
user should have access to ZK paths, for registry if nothing else, so this 
shouldn't be a problem


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java,
> >  line 61
> > <https://reviews.apache.org/r/46956/diff/2/?file=1387304#file1387304line61>
> >
> > I don't think we need to create a new instance of the 
> > ZKDelegationTokenSecretManager.
> > 
> > The one created earlier to generate tokens should be passed in.
> > 
> > The KeySigner could be an interface instead, and SecretManager (extends 
> > ZKDelegationTokenSecretManager) can implement this. ACL checks etc are 
> > already setup there. There's no requirement to have two independent copies 
> > of the ZKSecretManager running in the same daemon.
> 
> Sergey Shelukhin wrote:
> This one has completely different logic and even different template 
> parameter.
> 
> Siddharth Seth wrote:
> Most of the logic is in the BaseSecretManager itself, correct ?
> The same instance can be used to generate tokens, as well as sign. Is 
> there a downside to that ?
> Setting up two instances would create extra threads, and confusion while 
> debugging; Also potentially additional load to ZK, additional logins, etc

Merged the classes. They are created separately for now, I'll add central 
creation in one of the subsequent patches that adds llapcoordinator.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 21, 2016, 12:07 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 21, 2016, 12:07 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-s

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/
---

(Updated May 18, 2016, 8:36 p.m.)


Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 e28eddd 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
de817e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 03ee055 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
8abd198 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 279baf1 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 

Diff: https://reviews.apache.org/r/46956/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47546/
---

Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description
---


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 cffa493 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 

Diff: https://reviews.apache.org/r/47546/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

(Updated May 18, 2016, 11:04 p.m.)


Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
349ee14 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinatorImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
ce69ee6 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 e28eddd 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
349ee14 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinatorImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
de817e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 03ee055 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
8abd198 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 279baf1 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
ce69ee6 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: [VOTE] Apache Hive 2.0.1 Release Candidate 1

[Sergey Shelukhin]

With 4 votes the vote passes. I am waiting for the release to propagate to
mirrors before doing the remaining steps.

On 16/5/24, 10:38, "Owen O'Malley" <omal...@apache.org> wrote:

>+1
>* Verified the signatures
>* Signed Sergey's signature (and pushed to the keyservers)
>
>.. Owen
>
>
>On Mon, May 23, 2016 at 11:50 PM, Thejas Nair <thejas.n...@gmail.com>
>wrote:
>
>> +1
>> Also checked the release notes updates.
>>
>>
>> On Thu, May 12, 2016 at 12:28 PM, Alan Gates <alanfga...@gmail.com>
>>wrote:
>> > +1.  Checked the keys and signatures, did a full build with a clean
>> repo, checked for the presence of any binary files, checked the NOTICE
>>and
>> LICENSE files.
>> >
>> > Alan.
>> >
>> >> On May 3, 2016, at 21:25, Sergey Shelukhin <ser...@apache.org> wrote:
>> >>
>> >> Apache Hive 2.0.1 Release Candidate 1 is available here:
>> >>
>> >> http://home.apache.org/~sershe/hive-2.0.1-rc1/
>> >>
>> >>
>> >> Maven artifacts are available here:
>> >>
>> >> 
>>https://repository.apache.org/content/repositories/orgapachehive-1049/
>> >>
>> >>
>> >> Source tag for RC1 (github mirror) is:
>> >> https://github.com/apache/hive/releases/tag/release-2.0.1-rc1
>> >>
>> >>
>> >> Voting will conclude in 72 hours.
>> >>
>> >> Hive PMC Members: Please test and vote.
>> >>
>> >> Thanks.
>> >>
>> >>
>> >>
>> >
>>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/
---

(Updated May 21, 2016, 12:07 a.m.)


Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4cfa5f1 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 e28eddd 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
de817e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 03ee055 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
8abd198 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 279baf1 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 

Diff: https://reviews.apache.org/r/46956/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

 the same as the LLAP service 
> > user). (That needs to be validated)
> > 2. In case of external services - would this be the HS2 service user or 
> > the user associated with the external service ?
> > 
> > If it's the HS2 user each time, is the "user"/"realuser" field in the 
> > TokenIdentifier required ? That seems to be passed in as a null everywhere.
> > Assuming the appId is what will be used to differentiate different 
> > external clients ? and that in case of Tez - there's no differentiation.

This is the calling user in case of RPC.


> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java,
> >  line 61
> > <https://reviews.apache.org/r/46956/diff/2/?file=1387304#file1387304line61>
> >
> > I don't think we need to create a new instance of the 
> > ZKDelegationTokenSecretManager.
> > 
> > The one created earlier to generate tokens should be passed in.
> > 
> > The KeySigner could be an interface instead, and SecretManager (extends 
> > ZKDelegationTokenSecretManager) can implement this. ACL checks etc are 
> > already setup there. There's no requirement to have two independent copies 
> > of the ZKSecretManager running in the same daemon.

This one has completely different logic and even different template parameter.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
---


On May 18, 2016, 8:36 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 18, 2016, 8:36 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java,
> >  line 25
> > <https://reviews.apache.org/r/46956/diff/2/?file=1387299#file1387299line25>
> >
> > Think the patch which added Pair/ImmutablePair may have added a maven 
> > dependency. Should be removed if it was added explicitly for this.

it has since become necessary for StringUtils


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/#review134076
-------


On May 18, 2016, 8:36 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46956/
> ---
> 
> (Updated May 18, 2016, 8:36 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  e28eddd 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> de817e3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  b94fc2e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
>  03ee055 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  8abd198 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
>  eac0e8f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  74359fa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  279baf1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
>  762 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  a250882 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
> 
> Diff: https://reviews.apache.org/r/46956/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/
---

(Updated May 17, 2016, 9:58 p.m.)


Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 541af57 
  llap-client/pom.xml 4a75bbb 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
424769f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java 18355e6 
  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 cd11bdb 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 edf9b18 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
5731b2c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 db8bfa6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 f958bc4 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
c54e726 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java fd6465a 

Diff: https://reviews.apache.org/r/46579/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

> On May 17, 2016, 10:01 p.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 
> > 376
> > <https://reviews.apache.org/r/46579/diff/4-5/?file=1385106#file1385106line376>
> >
> > Required here as well ?

remote one doesn't start threads; also see the comment on the cache field


> On May 17, 2016, 10:01 p.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 
> > 347
> > <https://reviews.apache.org/r/46579/diff/4-5/?file=1385106#file1385106line347>
> >
> > null check required on the notification itself ?

not as far as I can tell. Evne the value check is just paranoid, it's probably 
unneeded


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/#review133639
-------


On May 17, 2016, 9:58 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46579/
> ---
> 
> (Updated May 17, 2016, 9:58 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
> Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 541af57 
>   llap-client/pom.xml 4a75bbb 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 424769f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java 18355e6 
>   llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
>  cd11bdb 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  edf9b18 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 5731b2c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  db8bfa6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  f958bc4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> c54e726 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java fd6465a 
> 
> Diff: https://reviews.apache.org/r/46579/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/
---

(Updated May 17, 2016, 12:56 a.m.)


Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 541af57 
  llap-client/pom.xml 4a75bbb 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
424769f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java 18355e6 
  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 cd11bdb 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 edf9b18 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
5731b2c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 db8bfa6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 f958bc4 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
c54e726 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java fd6465a 

Diff: https://reviews.apache.org/r/46579/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

> On May 17, 2016, 12:01 a.m., Siddharth Seth wrote:
> > llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java, line 24
> > <https://reviews.apache.org/r/46579/diff/3/?file=1374112#file1374112line24>
> >
> > Can appId be removed from the parameter list. Doesn't look like it's 
> > used anywhere.

it can be useful in future


> On May 17, 2016, 12:01 a.m., Siddharth Seth wrote:
> > llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java, line 57
> > <https://reviews.apache.org/r/46579/diff/3/?file=1374113#file1374113line57>
> >
> > Not used anywhere ?
> > Also, this ends up being very flaky - with the "org-apache-slider" 
> > string hardcoded there. Referencing the actual slider constant would be 
> > better.
> > Removing the unused code would be better though.

commented it out for now


> On May 17, 2016, 12:01 a.m., Siddharth Seth wrote:
> > llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java, line 61
> > <https://reviews.apache.org/r/46579/diff/3/?file=1374113#file1374113line61>
> >
> > Can this refer to a hardcoded string in the slider code base - very 
> > prone to breaking otherwise.

it's not accessible there; filed a slider jira. This method has been commented 
out


> On May 17, 2016, 12:01 a.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 
> > 353
> > <https://reviews.apache.org/r/46579/diff/3/?file=1374121#file1374121line353>
> >
> > Is this supposed to be creating the LocalClient each time (whcih 
> > creates the secret manager), or is the intent to eventually share the 
> > SecretManager ?
> > Similarly for the next line - create an instance of the factory each 
> > time, create client and then the token. The code explicitly mentions a new 
> > client being created for thread safety. Doesn't really matter if a new 
> > factory is created each time.
> > 
> > Looks like not sharing the SecretManager could be expensive given the 
> > KDC login each time ?

followed up in HIVE-13698


- Sergey


-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/#review133466
---


On May 5, 2016, 9:55 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46579/
> ---
> 
> (Updated May 5, 2016, 9:55 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
> Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bb74d99 
>   llap-client/pom.xml 50c06a4 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 424769f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java 18355e6 
>   llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
>  cd11bdb 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  edf9b18 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> e662de9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  db8bfa6 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  f958bc4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> c54e726 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java fd6465a 
> 
> Diff: https://reviews.apache.org/r/46579/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

> On May 3, 2016, 3:14 p.m., Siddharth Seth wrote:
> > Looks good in terms of functionality. (Ship it if you think the reflection 
> > is not super brittle :))
> > I think we should get rid of the reflection to access private methods ASAP 
> > though - it can be really brittle and cause difficult to debug failures 
> > with different hadoop versions - ideally before a 2.1 release.
> > 1) The perf issue may not be valid, and we could get away with using a 
> > single UGI.
> > 2) A UGI pool.
> > If this gets committed as is, could you please open a follow up blocker 
> > ticket for 2.1 to remove the reflection. I can take that up at a later 
> > point.

We'd only be able to remove reflection in newer versions of Hadoop, so it will 
still remain in the shim, at least.
There's unfortunately no way to clone the UGI (that I see).
The problem with pools or a single UGI is that we cannot add credentials. 
Therefore, LLAP won't work against anything other than HDFS (e.g. HBase, etc.) 
in this mode... Maybe there should be a config flag for the case of bugs. 
However, I think UGI state is fairly stable, esp. the ctor and the getters 
probably won't change. We should decide, cause if we don't want reflection 
there's no point in committing this for now.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/#review131507
-------


On May 2, 2016, 9:44 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46754/
> ---
> 
> (Updated May 2, 2016, 9:44 p.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2814353 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  6981061 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  3d45c7a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 63cb16b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  fcfa940 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> fea3dc7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  b3b571d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  76ba225 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  24f4442 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  08ee769 
> 
> Diff: https://reviews.apache.org/r/46754/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[VOTE] Apache Hive 2.0.1 Release Candidate 1

[Sergey Shelukhin]

Apache Hive 2.0.1 Release Candidate 1 is available here:

http://home.apache.org/~sershe/hive-2.0.1-rc1/


Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1049/


Source tag for RC1 (github mirror) is:
https://github.com/apache/hive/releases/tag/release-2.0.1-rc1


Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/
---

Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs
-

  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
424769f 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 e28eddd 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 edf9b18 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2bfe3ed 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
d23a44a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 db8bfa6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 03ee055 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
8abd198 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 3093de7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 f958bc4 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
c54e726 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 d699f20 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 

Diff: https://reviews.apache.org/r/46956/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46579/
---

(Updated May 5, 2016, 9:55 p.m.)


Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit 
Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bb74d99 
  llap-client/pom.xml 50c06a4 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
424769f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClientFactory.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java 18355e6 
  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java ce03de0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 cd11bdb 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 edf9b18 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
e662de9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 db8bfa6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 f958bc4 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
c54e726 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java fd6465a 

Diff: https://reviews.apache.org/r/46579/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: new HiveQA jenkins queue doesn't have JIRA numbers

[Sergey Shelukhin]

Thanks! That is, however, not very useful if you are trying to see if your
jira is somewhere in a 50-item queue ;)

On 16/4/29, 17:15, "Thejas Nair" <thejas.n...@gmail.com> wrote:

>the jira number shows up if you hover your mouse over the notepad-and-pen
>icon.
>
>
>On Thu, Apr 28, 2016 at 11:20 AM, Sergey Shelukhin
><ser...@hortonworks.com> wrote:
>> Subj. Is it a limitation of the new Jenkins instance, or can it be
>>enabled
>> again? The JIRA number associated with the run used to be shown for the
>> pending items in the left column.
>>
>

Re: Review Request 46809: HIVE-13442 LLAP: refactor submit API to be amenable to signing

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46809/
---

(Updated May 2, 2016, 11:01 p.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java ec6e439 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
d4cdac1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 3d45c7a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryFragmentInfo.java
 480a394 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
64c2b58 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
14657e6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 f621af2 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 fcfa940 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 24f4442 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 08ee769 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 a4f5d4d 

Diff: https://reviews.apache.org/r/46809/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

> On April 30, 2016, 7:06 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java,
> >  line 192
> > <https://reviews.apache.org/r/46754/diff/1/?file=1364102#file1364102line192>
> >
> > If we're using a common UGI across all tasks - which is the kerberos 
> > UGI - I don't think we should add these credentials. That'll end up leaking 
> > credentials across tasks - and in general is not required assuming SQL 
> > standard auth.
> > For non-sql standard auth - we could continue with credentials only.

What about HBase and other systems? using the keytab would mean giving that 
user access or not having it at all. I'll add the clone of UGI...


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/#review131247
-------


On April 27, 2016, 11:01 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46754/
> ---
> 
> (Updated April 27, 2016, 11:01 p.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5360ed4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
>  73f94f3 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  6af30d4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  e80fb15 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 33b41e8 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
>  e99e689 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  2a60123 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 6a72b4c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  b3b571d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  76ba225 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 8c7a539 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  24f4442 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  08ee769 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 8aca779 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 0584ad8 
> 
> Diff: https://reviews.apache.org/r/46754/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/
---

(Updated May 2, 2016, 9:44 p.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2814353 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 6981061 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 3d45c7a 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
63cb16b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 fcfa940 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
fea3dc7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 b3b571d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 24f4442 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 08ee769 

Diff: https://reviews.apache.org/r/46754/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46597/
---

(Updated May 2, 2016, 10:01 p.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4ab7b32 
  llap-common/src/java/org/apache/hadoop/hive/llap/DaemonId.java PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 23980d0 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 2e99a28 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 944c96c 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/ContainerRunner.java 
fc29371 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 3d45c7a 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
63cb16b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 dae1a3a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
64c2b58 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
14657e6 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/Scheduler.java 
26c8e55 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 f621af2 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 76ba225 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
8c7a539 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
610f266 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 24f4442 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 a65bf5c 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 3ea5ef9 

Diff: https://reviews.apache.org/r/46597/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/#review131384
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 
<https://reviews.apache.org/r/46754/#comment195333>

bogus


- Sergey Shelukhin


On May 2, 2016, 9:44 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46754/
> ---
> 
> (Updated May 2, 2016, 9:44 p.m.)
> 
> 
> Review request for hive and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2814353 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  6981061 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  3d45c7a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 63cb16b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  fcfa940 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> fea3dc7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  b3b571d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  76ba225 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  24f4442 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  08ee769 
> 
> Diff: https://reviews.apache.org/r/46754/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46754/
---

(Updated May 3, 2016, 9:07 p.m.)


Review request for hive and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/UgiFactory.java PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 06a6906 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 6981061 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 78b37f7 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
d23a44a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 fcfa940 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
fea3dc7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 b3b571d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 f958bc4 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 c6ba14e 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 08ee769 

Diff: https://reviews.apache.org/r/46754/diff/


Testing
---


Thanks,

Sergey Shelukhin