[jira] [Created] (HIVE-13922) Optimize the code path that analyzes/updates col stats

[Rajesh Balamohan (JIRA)]

Rajesh Balamohan created HIVE-13922:
---

 Summary: Optimize the code path that analyzes/updates col stats
 Key: HIVE-13922
 URL: https://issues.apache.org/jira/browse/HIVE-13922
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan
Priority: Minor


1. Depending on the number of partitions, 
HiveMetastore::update_partition_column_statistics::getPartValsFromName obtains 
the same table several times.

2. In ObjectStore, number of get calls to obtain the column stats can be 
considerably reduced when writing table/partition stats. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13921) Fix spark on yarn tests for HoS

[Rui Li (JIRA)]

Rui Li created HIVE-13921:
-

 Summary: Fix spark on yarn tests for HoS
 Key: HIVE-13921
 URL: https://issues.apache.org/jira/browse/HIVE-13921
 Project: Hive
  Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li


{{index_bitmap3}} and {{constprog_partitioner}} have been failing. Let's fix 
them here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13920) LLAP: pre-populate CVB cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-13920:
---

 Summary: LLAP: pre-populate CVB cache
 Key: HIVE-13920
 URL: https://issues.apache.org/jira/browse/HIVE-13920
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


To avoid allocations on the main path, we should pre-populate the CVB cache. 
The main difficulty is propagating column indexes earlier to allocate correct 
vectors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13919) TestHWISessionManager 'did not produce TEST-*.xml' is reported incorrectly

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-13919:
-

 Summary: TestHWISessionManager 'did not produce TEST-*.xml' is 
reported incorrectly
 Key: HIVE-13919
 URL: https://issues.apache.org/jira/browse/HIVE-13919
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


PTest reports this when 0 tests are run. In this case the Test file is 
completely ignored (@Ingore annotation) - and ptest ends up reporting it as not 
producing an XML file.
Either fix ptest to look at the skipped annotation, or disable the test in 
ptest friendly manner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13918) Towards a green jenkins precommit build

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-13918:
-

 Summary: Towards a green jenkins precommit build
 Key: HIVE-13918
 URL: https://issues.apache.org/jira/browse/HIVE-13918
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13917) Investigate and fix tests which are timing out in the precommit build

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-13917:
-

 Summary: Investigate and fix tests which are timing out in the 
precommit build
 Key: HIVE-13917
 URL: https://issues.apache.org/jira/browse/HIVE-13917
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Priority: Critical


Three tests seem to timeout consistently.
TestJdbcWithMiniHA
TestJdbcWithMiniMr
TestOperationLoggingAPIWithTez



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13916) Reduce the max runtime of a test further

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-13916:
-

 Summary: Reduce the max runtime of a test further
 Key: HIVE-13916
 URL: https://issues.apache.org/jira/browse/HIVE-13916
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


HIVE-13520 reduced this from 2h to 1h.

We should bring it down further - 40minutes.

>From logs added in HIVE-13507 - the following are the longest runtimes from 
>run 476.
{code}
 ElapsedTime(seconds)=979
 ElapsedTime(seconds)=1011
 ElapsedTime(seconds)=1014
 ElapsedTime(seconds)=1060
 ElapsedTime(seconds)=1114
 ElapsedTime(seconds)=1157
 ElapsedTime(seconds)=1169
 ElapsedTime(seconds)=1240
 ElapsedTime(seconds)=1307
 ElapsedTime(seconds)=1325
 ElapsedTime(seconds)=1326
 ElapsedTime(seconds)=1334
 ElapsedTime(seconds)=1368
 ElapsedTime(seconds)=1372
 ElapsedTime(seconds)=1395
 ElapsedTime(seconds)=1434
 ElapsedTime(seconds)=1459
 ElapsedTime(seconds)=1492
 ElapsedTime(seconds)=1499
 ElapsedTime(seconds)=1604
 ElapsedTime(seconds)=3601
 ElapsedTime(seconds)=3601
 ElapsedTime(seconds)=3601
{code}
The last 3 are tests that timed out. 
The longest valid test takes 1604 seconds or about 26 minutes. 1.5 times that 
is 40, which seems like a reasonable number.

With the current state of the builds - if the last 3 tests were not timing out 
- the build/test-run would complete 20minutes faster. 

There's two more stragglers before that - which add another 15 minutes to the 
test run. i.e. if we can fix some of these longer running tests, or have them 
trigger early - we can likely reduce the jenkins runtime further - 
TestParseNegative (1368s runtime) and TestHBaseMinimrCliDriver (1060seconds 
runtime)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13915) Avoid sending the VertexSpec along with each fragment in GenericUDTFGetSplits

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-13915:
-

 Summary: Avoid sending the VertexSpec along with each fragment in 
GenericUDTFGetSplits
 Key: HIVE-13915
 URL: https://issues.apache.org/jira/browse/HIVE-13915
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth


After changes in HIVE-13675, TaskSpec no longer needs to be constructed per 
split, and VertexSpec does not need to be sent over the wire for each split.

Considering that TaskSpec and VertexSpec can be fairly large - we should avoid 
sending them.

cc [~jdere], [~sershe]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135891
---


Ship it!




Looks good. I'll create a follow up to remove the unnecessary TaskSpec 
creation, and potentially skipping the unnecessary VertexSpec being sent each 
time as a response from GenericUDTFGetSplits.

- Siddharth Seth


On June 2, 2016, midnight, Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated June 2, 2016, midnight)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  af889b6 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> 478a40a 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 540f978 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  067a98e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 1df6df0 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  d439c07 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> 4174593 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  0420225 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 140dbda 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13914) Changing a column of orc table from float to double adds some fractional digits

[Takahiko Saito (JIRA)]

Takahiko Saito created HIVE-13914:
-

 Summary: Changing a column of orc table from float to double adds 
some fractional digits
 Key: HIVE-13914
 URL: https://issues.apache.org/jira/browse/HIVE-13914
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Takahiko Saito


Some fractional digits are added after altering a float column of an orc table 
to double:
{noformat}
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
No rows affected (0.233 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float) 
stored as orc;
No rows affected (0.095 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test 
values(-35664.76),(29497.34);
No rows affected (2.206 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
++--+
|   test.f   |
++--+
| -35664.76  |
| 29497.34   |
++--+
2 rows selected (0.131 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column f 
f double;
No rows affected (0.183 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
+--+--+
|  test.f  |
+--+--+
| -35664.76171875  |
| 29497.33984375   |
+--+--+
2 rows selected (0.114 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
{noformat}

The issue is NOT seen with text formatted table:
{noformat}
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float) ;
No rows affected (0.131 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test 
values(-35664.76),(29497.34);
No rows affected (2.114 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
++--+
|   test.f   |
++--+
| -35664.76  |
| 29497.34   |
++--+
2 rows selected (0.163 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> alter table test change column f 
f double;
No rows affected (0.184 seconds)
0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
++--+
|   test.f   |
++--+
| -35664.76  |
| 29497.34   |
++--+
2 rows selected (0.11 seconds)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13913) LLAP: introduce backpressure to recordreader

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-13913:
---

 Summary: LLAP: introduce backpressure to recordreader
 Key: HIVE-13913
 URL: https://issues.apache.org/jira/browse/HIVE-13913
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13912) DbTxnManager.commitTxn(): ORA-00918: column ambiguously defined

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-13912:
-

 Summary: DbTxnManager.commitTxn(): ORA-00918: column ambiguously 
defined
 Key: HIVE-13912
 URL: https://issues.apache.org/jira/browse/HIVE-13912
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.3.0, 2.1.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Priority: Blocker


{noformat}
Caused by: MetaException(message:Unable to update transaction database 
java.sql.SQLSyntaxErrorException: ORA-00918: column ambiguously defined

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
at 
oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:852)
at 
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at 
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at 
oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1477)
at 
oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:392)
at 
com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(TxnHandler.java:662)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.commit_txn(HiveMetaStore.java:5864)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy49.commit_txn(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.commitTxn(HiveMetaStoreClient.java:2090)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
at com.sun.proxy.$Proxy50.commitTxn(Unknown Source)
at 
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$SynchronizedMetaStoreClient.commitTxn(DbTxnManager.java:655)
at 
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:356)
at 
org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1024)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1321)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1071)
{noformat}

caused by 
{noformat}
(sqlGenerator.addLimitClause(1, "committed.ws_txnid, 
committed.ws_commit_id, committed.ws_database," +
  "committed.ws_table, committed.ws_partition, cur.ws_commit_id " +
  "from WRITE_SET committed INNER JOIN WRITE_SET cur " +
  "ON committed.ws_database=cur.ws_database and 
committed.ws_table=cur.ws_table " +
  //For partitioned table we always track writes at partition level 
(never at table)
  //and for non partitioned - always at table level, thus the same 
table should never
  //have entries with partition key and w/o
  "and (committed.ws_partition=cur.ws_partition or 
(committed.ws_partition is null and cur.ws_partition is null)) " +
  "where cur.ws_txnid <= committed.ws_commit_id" + //txns overlap; 
could replace ws_txnid
  // with txnid, though any decent DB should infer this
  " and cur.ws_txnid=" + txnid + //make sure RHS of join only has 
rows we just inserted as
  // part of this commitTxn() op
  " and committed.ws_txnid <> " + txnid + //and LHS only has 
committed txns
  //U+U and U+D is a conflict but D+D is not and we don't currently 
track I in WRITE_SET at all
  " and (committed.ws_operation_type=" + 
quoteChar(OpertaionType.UPDATE.sqlConst) +
  " OR 

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

(Updated June 2, 2016, midnight)


Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  
llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 af889b6 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
478a40a 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
540f978 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
 067a98e 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
1df6df0 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 d439c07 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
4174593 
  
llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
 0420225 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
140dbda 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 48159: HIVE-13901: Hivemetastore add partitions can be slow depending on filesystems

[Rajesh Balamohan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48159/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13901
https://issues.apache.org/jira/browse/HIVE-13901


Repository: hive-git


Description
---

Depending on FS, creating external tables & adding partitions can be expensive 
(e.g msck which adds all partitions).


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
94dd72e 

Diff: https://reviews.apache.org/r/48159/diff/


Testing
---


Thanks,

Rajesh Balamohan

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Sergey Shelukhin]

> On June 1, 2016, 10:14 p.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java,
> >  line 348
> > 
> >
> > I don't see where this is being enforced ? The signer is used 
> > irrespective of secure/non-secure.
> > 
> > Are we going with the approach of sending a serialized VertexSpec in 
> > non-secure (without a signatre), or sending the TaskSpec ?
> > 
> > Scratch that. Looks like it's being checked in createSignedVertexSpec. 
> > The comment still seems incorrect though, since TaskSpec is never sent any 
> > longer.
> > 
> > Assuming the serialized spec without a signature works in LLAP (i.e. 
> > when the daemon gets the VertexSpecBinary without a signature - it handles 
> > it correctly)

the comment is wrong


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135873
---


On June 1, 2016, 9:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated June 1, 2016, 9:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  af889b6 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> 478a40a 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 5aa4b84 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  067a98e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 1df6df0 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  d439c07 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> 4174593 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  0420225 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 140dbda 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13911) load inpath fails throwing org.apache.hadoop.security.AccessControlException

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-13911:


 Summary: load inpath fails throwing 
org.apache.hadoop.security.AccessControlException
 Key: HIVE-13911
 URL: https://issues.apache.org/jira/browse/HIVE-13911
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Attachments: HIVE-13911.1.patch

Similar to HIVE-13857



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 48158: HIVE-13909 upgrade ACLs in LLAP registry when the cluster is upgraded to secure

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48158/
---

Review request for hive, Gunther Hagleitner and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 5a9b24c 

Diff: https://reviews.apache.org/r/48158/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135873
---




llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java (line 44)


Remove TaskSpec from the cosntructor also, and a corresponding change in 
GenericUDTFGetSplits to construct TaskSpec only once if possible (otherwise 
that's a separate jira)



llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 (line 146)


Does this need to be volatile or an AtomicReference ?



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
(line 341)


I don't see where this is being enforced ? The signer is used irrespective 
of secure/non-secure.

Are we going with the approach of sending a serialized VertexSpec in 
non-secure (without a signatre), or sending the TaskSpec ?

Scratch that. Looks like it's being checked in createSignedVertexSpec. The 
comment still seems incorrect though, since TaskSpec is never sent any longer.

Assuming the serialized spec without a signature works in LLAP (i.e. when 
the daemon gets the VertexSpecBinary without a signature - it handles it 
correctly)


- Siddharth Seth


On June 1, 2016, 9:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated June 1, 2016, 9:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  af889b6 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> 478a40a 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 5aa4b84 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  067a98e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 1df6df0 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  d439c07 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> 4174593 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  0420225 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 140dbda 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Siddharth Seth]

- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135706
---


On June 1, 2016, 9:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated June 1, 2016, 9:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  af889b6 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> 478a40a 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 5aa4b84 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  067a98e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 1df6df0 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  d439c07 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> 4174593 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  0420225 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 140dbda 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13910) [Ranger-Hive] select from a table is not working if used as

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-13910:
--

 Summary: [Ranger-Hive] select from a table is not working if used 
as 
 Key: HIVE-13910
 URL: https://issues.apache.org/jira/browse/HIVE-13910
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


{code}
set hive.mapred.mode=nonstrict;
set 
hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;

create database newdb;

use newdb;

create table masking_test as select cast(key as int) as key, value from 
default.src;

use default;

explain select * from newdb.masking_test;
select * from newdb.masking_test;

explain select * from newdb.masking_test where key > 0;
select * from newdb.masking_test where key > 0;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Siddharth Seth]

> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 
> > 381
> > 
> >
> > Not sure why the user is required in the client. Seems like this will 
> > cause all kinds of issues with doAs=true.
> > Multiple llap instances is not supported yet - can avoid adding bits 
> > and pieces towards it.
> 
> Sergey Shelukhin wrote:
> cluster name in ZK paths/etc. includes user name... multiple LLAP 
> clusters are supported for different users. Anyway it's a simple thing

Any thoughts around doAs=true. Won't the ZK path constructed in this case be 
incorrect ?
That's officially not supported. Maybe we should have a check for this 
somewhere (not related to this patch) saying - If LLAP is being used, ensure 
doAs is false.


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135706
---


On June 1, 2016, 9:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated June 1, 2016, 9:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  af889b6 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> 478a40a 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 5aa4b84 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  067a98e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 1df6df0 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  d439c07 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> 4174593 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  0420225 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 140dbda 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13909) upgrade ACLs in LLAP registry when the cluster is upgraded to secure

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-13909:
---

 Summary: upgrade ACLs in LLAP registry when the cluster is 
upgraded to secure
 Key: HIVE-13909
 URL: https://issues.apache.org/jira/browse/HIVE-13909
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


ZK model has authentication and authorization mixed together, so it's 
impossible to set up acls that would carry over between unsecure and secure 
clusters in the normal case (i.e. work for a specific users no matter the 
authentication method).
To support cluster updates from unsecure to secure, we'd need to change the 
ACLs ourselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Siddharth Seth]

> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote:
> > llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java,
> >  line 350
> > 
> >
> > Is a signature always expected ? If so, may as well convert this to a 
> > precondition.
> > Otherwise, in what case will it not be expected ?
> 
> Sergey Shelukhin wrote:
> Unsecure case?

A comnment in the code mentions that TaskSpec will be used for the unsecure 
case. If we're using the vertexBinary for unsecure - that's even better, since 
it stays consistent. (There's a comment indicating the same somewhere else in 
the review)


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135706
---


On June 1, 2016, 9:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated June 1, 2016, 9:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  af889b6 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> 478a40a 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 5aa4b84 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  067a98e 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 1df6df0 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  d439c07 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> 4174593 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  0420225 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 140dbda 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-13908) Beeline adds extra fractional digits when you insert values to table with float data type

[Takahiko Saito (JIRA)]

Takahiko Saito created HIVE-13908:
-

 Summary: Beeline adds extra fractional digits when you insert 
values to table with float data type 
 Key: HIVE-13908
 URL: https://issues.apache.org/jira/browse/HIVE-13908
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Takahiko Saito


Via beeline, although -35664.76 is inserted,  -35664.76171875 is displayed
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table test;
No rows affected (0.067 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test(f float);
No rows affected (0.248 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test 
values(-35664.76),(29497.34);
INFO  : Tez session hasn't been created yet. Opening session
INFO  : Dag name: insert into table tes...35664.76),(29497.34)(Stage-1)
INFO  :

INFO  : Status: Running (Executing on YARN cluster with App id 
application_1464727816747_0019)

INFO  : Map 1: -/-
INFO  : Map 1: 0/1
INFO  : Map 1: 0/1
INFO  : Map 1: 0(+1)/1
INFO  : Map 1: 1/1
INFO  : Loading data to table default.test from 
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-01_20-16-32_885_9161749848563358684-1/-ext-1
INFO  : Table default.test stats: [numFiles=1, numRows=2, totalSize=19, 
rawDataSize=17]
No rows affected (31.725 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
+--+--+
|  test.f  |
+--+--+
| -35664.76171875  |
| 29497.33984375   |
+--+--+
2 rows selected (0.143 seconds)
{noformat}

The issue is not seen via Hive CLI:
{noformat}
hive> create table test(f float);
OK
Time taken: 0.32 seconds
hive> insert into table test values(-35664.76),(29497.34);
Query ID = hrt_qa_20160601202446_75f38c5d-f52b-45b3-b67a-8a8b0a194305
Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id 
application_1464727816747_0020)


VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

Map 1 ..   SUCCEEDED  1  100   0   0

VERTICES: 01/01  [==>>] 100%  ELAPSED TIME: 7.66 s

Loading data to table default.test
Table default.test stats: [numFiles=1, numRows=2, totalSize=19, rawDataSize=17]
OK
Time taken: 11.477 seconds
hive> select * from test;
OK
-35664.76
29497.34
Time taken: 0.144 seconds, Fetched: 2 row(s)
{noformat}

hdfs file shows expected value:
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> dfs -cat 
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/00_0
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> ;
+-+--+
| DFS Output  |
+-+--+
| -35664.76   |
| 29497.34|
+-+--+
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

(Updated June 1, 2016, 9:23 p.m.)


Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b2e6b6f 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  
llap-client/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 af889b6 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
478a40a 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
5aa4b84 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
 067a98e 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
1df6df0 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 d439c07 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
4174593 
  
llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
 0420225 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java d04cfa3 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
140dbda 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

[Sergey Shelukhin]

> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line 
> > 381
> > 
> >
> > Not sure why the user is required in the client. Seems like this will 
> > cause all kinds of issues with doAs=true.
> > Multiple llap instances is not supported yet - can avoid adding bits 
> > and pieces towards it.

cluster name in ZK paths/etc. includes user name... multiple LLAP clusters are 
supported for different users. Anyway it's a simple thing


> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java,
> >  line 398
> > 
> >
> > This seems like a guaranteed NPE. The user will be set in a proto - 
> > which will throw a NullPointerException.
> > 
> > Hopefully this will be caught by some tests.
> > 
> > Would be good to add a couple of tests here as well.

it's a magical UDTF living in HS2... kinda hard to unit test. We can file a 
jira to refactor


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/#review135706
---


On May 27, 2016, 9:32 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47563/
> ---
> 
> (Updated May 27, 2016, 9:32 p.m.)
> 
> 
> Review request for hive, Jason Dere and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
> submit
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a404bd 
>   llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 
> 6704294 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> 6c2618b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  f10351b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
> PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
>  PRE-CREATION 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> 465b204 
>   
> llap-common/src/java/org/apache/hadoop/hive/llap/security/SigningSecretManager.java
>  PRE-CREATION 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> e43b72b 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 349ee14 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 4306c22 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinatorImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  2524dc2 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
> PRE-CREATION 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java
>  PRE-CREATION 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  026df3b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 83d492a 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 
> 
> Diff: https://reviews.apache.org/r/47563/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Error trying to add a JIRA contributor

[Lefty Leverenz]

Thanks Lars.  That worked.

Excerpt from the HBase list (Sean Busbey-2):

If you paste in the full JIRA username  you'll get the "can't contact
server" message, but it will just be from the autocomplete. If you close it
and submit anyways, the user will be added.


-- Lefty


On Wed, Jun 1, 2016 at 2:26 AM, Lars Francke  wrote:

> The HBase community just discussed the same issue: <
>
> http://apache-hbase.679495.n3.nabble.com/Adding-JIRA-contributors-td4080220.html
> >
>
> In short: It only seems to time out getting the auto-complete data in the
> form. If you paste the full Jira Name and press Enter it should work.
>
> On Wed, Jun 1, 2016 at 3:55 AM, Lefty Leverenz 
> wrote:
>
> > Ashutosh, are you still able to add JIRA contributors?  Both Sushanth
> and I
> > got this error repeatedly when we tried to add Shannon Ladymon to the
> list:
> >
> > "The JIRA server could not be contacted. This may be a temporary glitch
> or
> > the server may be down."
> >
> >
> > And yet I am able to add her to the committer or administrator list
> > (discovered by accident), so the JIRA server is not down.  Is this due
> to a
> > limit on the number of contributors?
> >
> > -- Lefty
> >
>

[jira] [Created] (HIVE-13907) TestCliDriver.stats_list_bucket.q fails due to JDK8 order issues

[JIRA]

Sergio Peña created HIVE-13907:
--

 Summary: TestCliDriver.stats_list_bucket.q fails due to JDK8 order 
issues
 Key: HIVE-13907
 URL: https://issues.apache.org/jira/browse/HIVE-13907
 Project: Hive
  Issue Type: Bug
Reporter: Sergio Peña
Assignee: Mohit Sabharwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13906) Remove guava dependence from storage-api module

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-13906:


 Summary: Remove guava dependence from storage-api module
 Key: HIVE-13906
 URL: https://issues.apache.org/jira/browse/HIVE-13906
 Project: Hive
  Issue Type: Bug
  Components: storage-api
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Guava is a very problematic library to depend on because of the version 
incompatibilities and the use of it in the storage-api module causes it to leak 
into everything that depends on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13905) optimize ColumnStatsTask::constructColumnStatsFromPackedRows to have lesser number of getTable calls

[Rajesh Balamohan (JIRA)]

Rajesh Balamohan created HIVE-13905:
---

 Summary: optimize 
ColumnStatsTask::constructColumnStatsFromPackedRows to have lesser number of 
getTable calls
 Key: HIVE-13905
 URL: https://issues.apache.org/jira/browse/HIVE-13905
 Project: Hive
  Issue Type: Improvement
  Components: Query Planning
Reporter: Rajesh Balamohan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13904) Ignore case when retrieving ColumnInfo from RowResolver

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-13904:
--

 Summary: Ignore case when retrieving ColumnInfo from RowResolver
 Key: HIVE-13904
 URL: https://issues.apache.org/jira/browse/HIVE-13904
 Project: Hive
  Issue Type: Bug
  Components: Parser
Affects Versions: 2.0.1, 2.1.0, 2.2.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48118: HIVE-13903: getFunctionInfo is downloading jar on every call

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48118/#review135781
---


Ship it!




Ship It!

- Amareshwari Sriramadasu


On June 1, 2016, 9:44 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48118/
> ---
> 
> (Updated June 1, 2016, 9:44 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13903
> https://issues.apache.org/jira/browse/HIVE-13903
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java 
> 86df74d26ddd60736214c0f9acb835f4253a2c17 
> 
> Diff: https://reviews.apache.org/r/48118/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

Review Request 48118: HIVE-13903: getFunctionInfo is downloading jar on every call

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48118/
---

Review request for hive.


Bugs: HIVE-13903
https://issues.apache.org/jira/browse/HIVE-13903


Repository: hive-git


Description
---

on queries using permanent udfs, the jar file of the udf is downloaded multiple 
times. Each call originating from Registry.getFunctionInfo. This increases time 
for the query, especially if that query is just an explain query. The jar 
should be downloaded once, and not downloaded again if the udf class is 
accessible in the current thread.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java 
86df74d26ddd60736214c0f9acb835f4253a2c17 

Diff: https://reviews.apache.org/r/48118/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Created] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Rajat Khandelwal (JIRA)]

Rajat Khandelwal created HIVE-13903:
---

 Summary: getFunctionInfo is downloading jar on every call
 Key: HIVE-13903
 URL: https://issues.apache.org/jira/browse/HIVE-13903
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Error trying to add a JIRA contributor

[Lars Francke]

The HBase community just discussed the same issue: <
http://apache-hbase.679495.n3.nabble.com/Adding-JIRA-contributors-td4080220.html
>

In short: It only seems to time out getting the auto-complete data in the
form. If you paste the full Jira Name and press Enter it should work.

On Wed, Jun 1, 2016 at 3:55 AM, Lefty Leverenz 
wrote:

> Ashutosh, are you still able to add JIRA contributors?  Both Sushanth and I
> got this error repeatedly when we tried to add Shannon Ladymon to the list:
>
> "The JIRA server could not be contacted. This may be a temporary glitch or
> the server may be down."
>
>
> And yet I am able to add her to the committer or administrator list
> (discovered by accident), so the JIRA server is not down.  Is this due to a
> limit on the number of contributors?
>
> -- Lefty
>

[jira] [Created] (HIVE-13902) [Refactor] Minimize metastore jar dependencies on task nodes

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-13902:
---

 Summary: [Refactor] Minimize metastore jar dependencies on task 
nodes
 Key: HIVE-13902
 URL: https://issues.apache.org/jira/browse/HIVE-13902
 Project: Hive
  Issue Type: Improvement
  Components: Metastore, Query Processor
Affects Versions: 2.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Unnecessary dependency on metastore jar can be eliminated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)