[jira] [Created] (HIVE-12591) LLAP cache counters displays -ve value for CacheCapacityUsed

2015-12-03 Thread Rajesh Balamohan (JIRA) 
Rajesh Balamohan created HIVE-12591:
---

 Summary: LLAP cache counters displays -ve value for 
CacheCapacityUsed
 Key: HIVE-12591
 URL: https://issues.apache.org/jira/browse/HIVE-12591
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Rajesh Balamohan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40948: HIVE-12499 : Add HMS metrics for number of tables and partitions

2015-12-03 Thread Lenni Kuff 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40948/#review108948
---



itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreMetrics.java
 (line 104)


maybe add a partitioned/unpartitioned table that is in a different database 
for extra test coverage



metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java (line 
414)


Would it make sense to use a cached-gauge for these operations? 

https://dropwizard.github.io/metrics/3.1.0/manual/core/#cached-gauges



metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java (line 
5777)


How expensive is this? What are your thoughts on doing this nce at startup, 
then increment/decrementing as individual objects are added/removed?


- Lenni Kuff


On Dec. 4, 2015, 1:51 a.m., Szehon Ho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40948/
> ---
> 
> (Updated Dec. 4, 2015, 1:51 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12499
> https://issues.apache.org/jira/browse/HIVE-12499
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add separate timer thread that polls for count of database, table, partition 
> entries to publish as metrics, the period is configurable.  Delay in getting 
> exact number should be ok as this is for monitoring.
> 
> Implemented for HBase and DB metastores.
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
>  95e2bcf 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4d881ba 
>   common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java 
> fd420f7 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreMetrics.java
>  f571c7c 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseMetastoreMetrics.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 00602e1 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 1c0ab6d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5b36b03 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
> 2fb3e8f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 98e6c75 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  9a1d159 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  8dde0af 
> 
> Diff: https://reviews.apache.org/r/40948/diff/
> 
> 
> Testing
> ---
> 
> Added unit tests for HBase and Db metastores.
> 
> 
> Thanks,
> 
> Szehon Ho
> 
>

Re: Review Request 40365: HIVE-11358 Move LlapConfiguration into HiveConf and document the settings

2015-12-03 Thread Lefty Leverenz 


> On Dec. 1, 2015, 8:34 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 2460-2470
> > 
> >
> > [..].connection.timeout.ms:  Is "before" correct in "before a failure" 
> > or should it be "Timeout (in milliseconds) after a connection failure ..."?
> > 
> > Hm, I must be confused because the next parameter seems to cover that 
> > case -- [..].sleep.between.retries.
> 
> Lefty Leverenz wrote:
> I still don't understand what the task communicator connection timeout is 
> (or how you'd have a timeout _before_ a failure), but if you're satisfied 
> with the current description then it's okay with me.

Sergey explained in the JIRA comments:  "One is the standard connection timeout 
before we give up and there's a failure, the other is a wait interval before 
retrying."  Now I understand.


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40365/#review108464
---


On Dec. 2, 2015, 3:41 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40365/
> ---
> 
> (Updated Dec. 2, 2015, 3:41 a.m.)
> 
> 
> Review request for hive, Lefty Leverenz and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
>   data/conf/llap/llap-daemon-site.xml f2851a7 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/configuration/LlapConfiguration.java
>  bd09024 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  34e0682 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  a8e1465 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
>  d474b6f 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
> 317fa20 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java 
> 6d54fd4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  4b28b53 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryFileCleaner.java
>  bc18a77 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  2db2833 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  9520413 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/java/org/apache/tez/dag/app/rm/LlapTaskSchedulerService.java 
> e920f86 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 4525ab9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   
> llap-server/src/test/org/apache/tez/dag/app/rm/TestLlapTaskSchedulerService.java
>  23724a4 
> 
> Diff: https://reviews.apache.org/r/40365/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12592) Expose connection pool tuning props in TxnHandler

2015-12-03 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-12592:
-

 Summary: Expose connection pool tuning props in TxnHandler
 Key: HIVE-12592
 URL: https://issues.apache.org/jira/browse/HIVE-12592
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


BoneCP allows various pool tuning options like connection timeout, num 
connections, etc

There should be a config based way to set these



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40898: HIVE-12431: Support timeout for global compile lock

2015-12-03 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40898/
---

(Updated Dec. 3, 2015, 8:14 a.m.)


Review request for hive.


Bugs: HIVE-12431
https://issues.apache.org/jira/browse/HIVE-12431


Repository: hive-git


Description
---

HIVE-12431: Support timeout for global compile lock

When global (HS2-wide) compile lock is configured, a long-compiling request 
will block remaining sessions indefinitely. 

This patch allows the user to configure the maximum time a request will wait
to acquire the compile lock. Note that this configuration does not apply when
session scoped compile locking is configured.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
7f9607129eb1f5f43e8a728cf7d2a56c1ed5af49 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
62b608cbf53c371d1743df40988daf85f76a0867 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
8a47605630066e39272f506c6e309b108b8455dd 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
d90002bd16e46b5ce970d4c6c544a9c7605328d1 

Diff: https://reviews.apache.org/r/40898/diff/


Testing
---

TestEmbeddedThriftBinaryCLIService#testGlobalCompileLockTimeout


Thanks,

Mohit Sabharwal

[jira] [Created] (HIVE-12582) fix hiveserver2.sh script like metastore.sh is.

2015-12-03 Thread Stanilovsky Evgeny (JIRA) 
Stanilovsky Evgeny created HIVE-12582:
-

 Summary: fix hiveserver2.sh script like metastore.sh is.
 Key: HIVE-12582
 URL: https://issues.apache.org/jira/browse/HIVE-12582
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
 Environment: linux
Reporter: Stanilovsky Evgeny
Assignee: Vaibhav Gumashta
Priority: Trivial
 Fix For: 1.0.0


in case of altering startup jvm properties, it would be great to fix 
hiveserver2.sh, appending : export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS 
$HADOOP_OPTS" like it contains in metastore.sh



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 40898: HIVE-12431: Support timeout for global compile lock

2015-12-03 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40898/
---

Review request for hive.


Bugs: HIVE-12431
https://issues.apache.org/jira/browse/HIVE-12431


Repository: hive-git


Description
---

HIVE-12431: Support timeout for global compile lock

When global (HS2-wide) compile lock is configured, a long-compiling request 
will block remaining sessions indefinitely. 

This patch allows the user to configure the maximum time a request will wait
to acquire the compile lock. Note that this configuration does not apply when
session scoped compile locking is configured.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
7f9607129eb1f5f43e8a728cf7d2a56c1ed5af49 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
62b608cbf53c371d1743df40988daf85f76a0867 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
8a47605630066e39272f506c6e309b108b8455dd 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
d90002bd16e46b5ce970d4c6c544a9c7605328d1 

Diff: https://reviews.apache.org/r/40898/diff/


Testing
---

TestEmbeddedThriftBinaryCLIService#testGlobalCompileLockTimeout


Thanks,

Mohit Sabharwal

Re: Review Request 40898: HIVE-12431: Support timeout for global compile lock

2015-12-03 Thread Lenni Kuff 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40898/#review108847
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 1849)


Curious - does it make sense to only apply this to the "global" compile 
lock? Wouldn't this also be applicable for the session-level compile lock?



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 139)


Is there a better way we can add in these test hook (create a mock driver 
or something)?



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1265)


Add a comment to this function to describe what it does and info about the 
return value.

Maybe rename to "tryAcquireCompileLock"?



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1277)


Should this be INFO level? It might be useful to log the query along with 
this message for debug purposes.



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1280)


Can we include the query text here?


- Lenni Kuff


On Dec. 3, 2015, 8:14 a.m., Mohit Sabharwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40898/
> ---
> 
> (Updated Dec. 3, 2015, 8:14 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12431
> https://issues.apache.org/jira/browse/HIVE-12431
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12431: Support timeout for global compile lock
> 
> When global (HS2-wide) compile lock is configured, a long-compiling request 
> will block remaining sessions indefinitely. 
> 
> This patch allows the user to configure the maximum time a request will wait
> to acquire the compile lock. Note that this configuration does not apply when
> session scoped compile locking is configured.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 7f9607129eb1f5f43e8a728cf7d2a56c1ed5af49 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 62b608cbf53c371d1743df40988daf85f76a0867 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
> 8a47605630066e39272f506c6e309b108b8455dd 
>   service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
> d90002bd16e46b5ce970d4c6c544a9c7605328d1 
> 
> Diff: https://reviews.apache.org/r/40898/diff/
> 
> 
> Testing
> ---
> 
> TestEmbeddedThriftBinaryCLIService#testGlobalCompileLockTimeout
> 
> 
> Thanks,
> 
> Mohit Sabharwal
> 
>

Re: Review Request 40887: HIVE-12574: windowing function returns incorrect result when the window size is larger than the partition size

2015-12-03 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40887/
---

(Updated Dec. 3, 2015, 4:06 p.m.)


Review request for hive.


Changes
---

Include the changes for count() and collect_set() fix.


Repository: hive-git


Description
---

HIVE-12574: windowing function returns incorrect result when the window size is 
larger than the partition size


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 9caffb6 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFFirstValue.java 
160ce91 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLastValue.java 
f917621 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java 98abd5c 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStreamingEvaluator.java
 3c76404 
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 
9d21103 
  ql/src/test/queries/clientpositive/windowing_windowspec4.q PRE-CREATION 
  ql/src/test/results/clientpositive/windowing_windowspec4.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/40887/diff/


Testing
---


Thanks,

Aihua Xu

[jira] [Created] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance

2015-12-03 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-12583:
-

 Summary: HS2 ShutdownHookManager holds extra of Driver instance 
 Key: HIVE-12583
 URL: https://issues.apache.org/jira/browse/HIVE-12583
 Project: Hive
  Issue Type: Bug
  Components: Locking
Affects Versions: 1.3.0
Reporter: Daniel Dai
Assignee: Daniel Dai


HIVE-12266 add a shutdown hook for every Driver instance to release the lock th 
session holds in case Driver does not exist elegantly. However, that holds all 
Driver instances and HS2 may run out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12584) Vectorized join with partition column of type char does not trim spaces

2015-12-03 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-12584:


 Summary: Vectorized join with partition column of type char does 
not trim spaces 
 Key: HIVE-12584
 URL: https://issues.apache.org/jira/browse/HIVE-12584
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 1.3.0, 2.0.0, 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Priority: Critical


When a table is partitioned on a column of type char and if join is performed 
on partitioned column then following exception gets thrown from hashtable loader

{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized 
to 5
at 
org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:82)
at 
org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:92)
... 4 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized 
to 5
at 
org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:216)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
at 
org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75)
... 5 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 
reserialized to 5
at 
org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.sanityCheckKeyForTag(MapJoinBytesTableContainer.java:276)
at 
org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.getHashFromKey(MapJoinBytesTableContainer.java:247)
at 
org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.internalPutRow(HybridHashTableContainer.java:451)
at 
org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.putRow(HybridHashTableContainer.java:444)
at 
org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:210)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108891
---



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 480)


this seems similar with the code in SQLOperation, perhaps they can be 
refactored into an utility method used by both



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 508)


please log instead



service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java (line 
511)


nit: IIRC setting it to null is also harmless


- Sergey Shelukhin


On Dec. 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40898: HIVE-12431: Support timeout for global compile lock

2015-12-03 Thread Mohit Sabharwal 


> On Dec. 3, 2015, 5:40 p.m., Lenni Kuff wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1849
> > 
> >
> > Curious - does it make sense to only apply this to the "global" compile 
> > lock? Wouldn't this also be applicable for the session-level compile lock?

Yeah, makes sense. Changed it to apply to either case.


> On Dec. 3, 2015, 5:40 p.m., Lenni Kuff wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 139
> > 
> >
> > Is there a better way we can add in these test hook (create a mock 
> > driver or something)?

Mocking the driver to test concurrent requests is a bit tricky. Cleaned this up 
by adding a compile time hook instead as suggesed by Sergey.


> On Dec. 3, 2015, 5:40 p.m., Lenni Kuff wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1265
> > 
> >
> > Add a comment to this function to describe what it does and info about 
> > the return value.
> > 
> > Maybe rename to "tryAcquireCompileLock"?

Fixed.


> On Dec. 3, 2015, 5:40 p.m., Lenni Kuff wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1277
> > 
> >
> > Should this be INFO level? It might be useful to log the query along 
> > with this message for debug purposes.

Done.


> On Dec. 3, 2015, 5:40 p.m., Lenni Kuff wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1280
> > 
> >
> > Can we include the query text here?

Done.


- Mohit


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40898/#review108847
---


On Dec. 3, 2015, 8:14 a.m., Mohit Sabharwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40898/
> ---
> 
> (Updated Dec. 3, 2015, 8:14 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12431
> https://issues.apache.org/jira/browse/HIVE-12431
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12431: Support timeout for global compile lock
> 
> When global (HS2-wide) compile lock is configured, a long-compiling request 
> will block remaining sessions indefinitely. 
> 
> This patch allows the user to configure the maximum time a request will wait
> to acquire the compile lock. Note that this configuration does not apply when
> session scoped compile locking is configured.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 7f9607129eb1f5f43e8a728cf7d2a56c1ed5af49 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 62b608cbf53c371d1743df40988daf85f76a0867 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
> 8a47605630066e39272f506c6e309b108b8455dd 
>   service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
> d90002bd16e46b5ce970d4c6c544a9c7605328d1 
> 
> Diff: https://reviews.apache.org/r/40898/diff/
> 
> 
> Testing
> ---
> 
> TestEmbeddedThriftBinaryCLIService#testGlobalCompileLockTimeout
> 
> 
> Thanks,
> 
> Mohit Sabharwal
> 
>

Re: Review Request 40898: HIVE-12431: Support timeout for compile lock

2015-12-03 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40898/
---

(Updated Dec. 3, 2015, 11:04 p.m.)


Review request for hive.


Changes
---

Incorp fb.


Summary (updated)
-

HIVE-12431: Support timeout for compile lock


Bugs: HIVE-12431
https://issues.apache.org/jira/browse/HIVE-12431


Repository: hive-git


Description (updated)
---

HIVE-12431: Support timeout for compile lock

When compile lock is configured, a long-compiling request will block 
remaining sessions or concurrent requests for a given session indefinitely. 

This patch allows the user to configure the maximum time a request will wait
to acquire the compile lock.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
7f9607129eb1f5f43e8a728cf7d2a56c1ed5af49 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
62b608cbf53c371d1743df40988daf85f76a0867 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
8a47605630066e39272f506c6e309b108b8455dd 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
d90002bd16e46b5ce970d4c6c544a9c7605328d1 

Diff: https://reviews.apache.org/r/40898/diff/


Testing
---

TestEmbeddedThriftBinaryCLIService#testGlobalCompileLockTimeout


Thanks,

Mohit Sabharwal

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Dec. 3, 2015, 11:07 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f96071 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java 
f116de4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 34e0682 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
 d474b6f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
4525ab9 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 35868: HIVE-11107

2015-12-03 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35868/
---

(Updated Dec. 3, 2015, 9:48 p.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Changes
---

Integrated Ashutosh's comments from the previous upload.


Repository: hive-git


Description
---

Support for Performance regression test suite with TPCDS


Diffs (updated)
-

  data/conf/perf-reg/hive-site.xml PRE-CREATION 
  data/conf/perf-reg/tez-site.xml PRE-CREATION 
  data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt PRE-CREATION 
  data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt PRE-CREATION 
  data/files/tpcds-perf/metastore_export/sql/TAB_COL_STATS.sql PRE-CREATION 
  data/scripts/q_perf_test_init.sql PRE-CREATION 
  itests/qtest/pom.xml 8f6807a 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6713a2f 
  ql/src/test/queries/clientpositive/perf/query12.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query13.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query15.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query17.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query18.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query19.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query20.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query22.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query25.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query26.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query27.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query29.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query31.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query34.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query39.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query40.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query42.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query43.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query45.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query46.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query48.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query50.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query52.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query54.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query55.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query64.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query68.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query7.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query70.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query71.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query72.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query73.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query75.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query76.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query79.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query80.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query82.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query84.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query85.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query87.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query90.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query91.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query92.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query93.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query94.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query95.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query96.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query97.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query98.q PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query12.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query13.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query15.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query17.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query18.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query19.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query20.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query22.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query25.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query26.q.out

[jira] [Created] (HIVE-12585) fix TxnHandler connection leak

2015-12-03 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-12585:
-

 Summary: fix TxnHandler connection leak
 Key: HIVE-12585
 URL: https://issues.apache.org/jira/browse/HIVE-12585
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.3.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


checkLock(CheckLockRequest rqst) is leaking connection



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2015-12-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108889
---



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 390)


some existing configuration is probably needed (and better)



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 401)


I am not sure how the original rowset works; this may not be good for large 
result sets. Should be ok as long as we have parity with the original, but 
original has fetchSize parameter which I assume could be doing some batching. 
That might be useful to have



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 402)


also, I wonder if it makes sense to add rows to rowset immediately after 
reading. Storing all rows and then adding them all to rowset stores all rows 
twice in memory.



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 455)


nit: why object?



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 465)


I wonder why we need to have thrift serialization? One of the goals is to 
avoid it. Perhaps it can be done in a follow-up JIRA



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 485)


nit: useless check, schema is already used above to get descriptors



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 491)


nit: get(pos) can be done once



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 506)


again please use existing conf object


- Sergey Shelukhin


On Dec. 2, 2015, 12:52 p.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 2, 2015, 12:52 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 180f99e8 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 8fafd61 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 25cefc2 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 1331a99 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> a14908b 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

[jira] [Created] (HIVE-12587) Support to add partitioned data set to TestPerfCliDriver

2015-12-03 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-12587:


 Summary: Support to add partitioned data set to TestPerfCliDriver
 Key: HIVE-12587
 URL: https://issues.apache.org/jira/browse/HIVE-12587
 Project: Hive
  Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 35868: HIVE-11107

2015-12-03 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35868/
---

(Updated Dec. 4, 2015, 12:05 a.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Changes
---

Added the missing queries to the TPCDS.


Repository: hive-git


Description
---

Support for Performance regression test suite with TPCDS


Diffs (updated)
-

  data/conf/perf-reg/hive-site.xml PRE-CREATION 
  data/conf/perf-reg/tez-site.xml PRE-CREATION 
  data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt PRE-CREATION 
  data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt PRE-CREATION 
  data/files/tpcds-perf/metastore_export/sql/TAB_COL_STATS.sql PRE-CREATION 
  data/scripts/q_perf_test_init.sql PRE-CREATION 
  itests/qtest/pom.xml 8f6807a 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6713a2f 
  ql/src/test/queries/clientpositive/perf/query12.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query13.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query15.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query17.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query18.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query19.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query20.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query21.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query22.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query25.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query26.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query27.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query28.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query29.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query31.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query32.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query34.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query39.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query40.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query42.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query43.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query45.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query46.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query48.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query50.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query51.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query52.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query54.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query55.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query56.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query58.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query60.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query64.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query65.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query66.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query67.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query68.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query7.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query70.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query71.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query72.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query73.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query75.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query76.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query79.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query80.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query82.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query84.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query85.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query87.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query88.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query89.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query90.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query91.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query92.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query93.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query94.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query95.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query96.q PRE-CREATION

[jira] [Created] (HIVE-12586) Support for Performance regression test suite

2015-12-03 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-12586:


 Summary: Support for Performance regression test suite
 Key: HIVE-12586
 URL: https://issues.apache.org/jira/browse/HIVE-12586
 Project: Hive
  Issue Type: Task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


Umbrella jira to add TestPerfCliDriver to the existing test framework



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12588) Support to use run TestPerfCliDriver test suite using HBase metastore db

2015-12-03 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-12588:


 Summary: Support to use run TestPerfCliDriver test suite using 
HBase metastore db
 Key: HIVE-12588
 URL: https://issues.apache.org/jira/browse/HIVE-12588
 Project: Hive
  Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12589) LLAP service gen is broken for log4j2

2015-12-03 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12589:
---

 Summary: LLAP service gen is broken for log4j2
 Key: HIVE-12589
 URL: https://issues.apache.org/jira/browse/HIVE-12589
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12590) Repeated UDAFs with literals can produce incorrect result

2015-12-03 Thread Laljo John Pullokkaran (JIRA) 
Laljo John Pullokkaran created HIVE-12590:
-

 Summary: Repeated UDAFs with literals can produce incorrect result
 Key: HIVE-12590
 URL: https://issues.apache.org/jira/browse/HIVE-12590
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 1.2.1, 1.1.1, 1.0.1, 2.0.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Priority: Critical


Repeated UDAF with literals could produce wrong result.
This is not a common use case, nevertheless a bug.

hive> select max('pants'), max('pANTS') from t1 group by key;
 Total MapReduce CPU Time Spent: 0 msec
OK
pANTS   pANTS
pANTS   pANTS
pANTS   pANTS
pANTS   pANTS
pANTS   pANTS
Time taken: 296.252 seconds, Fetched: 5 row(s)






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 40948: HIVE-12499 : Add HMS metrics for number of tables and partitions

2015-12-03 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40948/
---

Review request for hive.


Bugs: HIVE-12499
https://issues.apache.org/jira/browse/HIVE-12499


Repository: hive-git


Description
---

Add separate timer thread that polls for count of database, table, partition 
entries to publish as metrics, the period is configurable.  Delay in getting 
exact number should be ok as this is for monitoring.

Implemented for HBase and DB metastores.


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
 95e2bcf 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4d881ba 
  common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java 
fd420f7 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreMetrics.java
 f571c7c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseMetastoreMetrics.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
00602e1 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1c0ab6d 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5b36b03 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
2fb3e8f 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
98e6c75 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 9a1d159 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 8dde0af 

Diff: https://reviews.apache.org/r/40948/diff/


Testing
---

Added unit tests for HBase and Db metastores.


Thanks,

Szehon Ho