[jira] [Created] (HIVE-18953) Implement CHECK constraint

2018-03-13 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-18953:
--

 Summary: Implement CHECK constraint
 Key: HIVE-18953
 URL: https://issues.apache.org/jira/browse/HIVE-18953
 Project: Hive
  Issue Type: New Feature
Reporter: Vineet Garg
Assignee: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 


> On March 13, 2018, 8:41 p.m., Sergey Shelukhin wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
> > Lines 123 (patched)
> > 
> >
> > nit: why is a thread needed when we just join immediately?

good catch.. join() supposed to happen together for the different threads.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199112
---


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 


> On March 13, 2018, 9:03 p.m., Sergey Shelukhin wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java
> > Lines 663 (patched)
> > 
> >
> > should this be addressed? the starting sessionstate will not be present 
> > iirc

Fixed.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199114
---


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199139
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java
Line 55 (original), 56 (patched)


Moved the casts internally to getInstances()



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 198 (patched)


Fixed.



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 202 (patched)


Good point. It is possible to lose leadership just before publishing it. 
Added a lock to protect around races. also reordered to move hasLeadership 
happen just before publishing endpoint.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 666 (patched)


HIVE-18952 created



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 670 (patched)


Sorry bad rebase. Fixed it.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 707 (patched)


Fixed it.


- Prasanth_J


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>

[jira] [Created] (HIVE-18952) Tez session disconnect and reconnect on HS2 HA failover

2018-03-13 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-18952:


 Summary: Tez session disconnect and reconnect on HS2 HA failover
 Key: HIVE-18952
 URL: https://issues.apache.org/jira/browse/HIVE-18952
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


Now that TEZ-3892 is committed, HIVE-18281 can make use of tez session 
disconnect and reconnect on HA failover.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199136
---




llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
Line 330 (original), 330 (patched)


Fixed method signature. A path can have only one instance.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Lines 67 (patched)


UUID is static. This is for the unit test.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 84 (patched)


these are defaults. llap and tez AM uses the defaults but HS2 will override 
or ignore some prefixes.. need c'tor override for all prefixes.

LLAP path:
/llap-sasl/hiveuser/hostname/workers/worker-000

HS2 path:
/hs2ActivePassiveHA-sasl/instances/instance-00



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 168 (patched)


This is required for ZK based service discovery. Clients connecting using 
ZK based service discovery to HS2 will have namespace in the URL (cannot have 
subpaths like /ns/user/workers). 

JDBC URL will have hs2ActivePassiveHA as namespace.

For HS2 the ACL provider path will be
/hs2ActivePassiveHA-sasl/instances/

There is no per user restriction (JDBC URL doesn't get users).



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 406 (original), 459 (patched)


IntelliJ rule violation :) final is unnecessary for static methods.



llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java
Lines 27 (patched)


llap-common is not dependent on llap-client. Dropped..



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Line 1365 (original), 1375 (patched)


ServiceRegistry interface had LlapServiceInstanceSet which is specific to 
LLAP. ServiceRegistry is updated to return generic ServiceInstanceSet. 
HiveServer2HAInstanceSet extends it to add isLeader() interface on top of 
ServiceRegistry.


- Prasanth_J


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
>

[jira] [Created] (HIVE-18951) Fix the llapdump usage error in llapdump.sh

2018-03-13 Thread Saijin Huang (JIRA) 
Saijin Huang created HIVE-18951:
---

 Summary: Fix the llapdump usage error in llapdump.sh
 Key: HIVE-18951
 URL: https://issues.apache.org/jira/browse/HIVE-18951
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Saijin Huang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66041: [HIVE-18343] Remove LinkedList from ColumnStatsSemanticAnalyzer.java

2018-03-13 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66041/#review199135
---


Ship it!




Ship It!

- Sahil Takiar


On March 14, 2018, 12:17 a.m., David Mollitor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66041/
> ---
> 
> (Updated March 14, 2018, 12:17 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> [HIVE-18343] Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> c97e2a9 
> 
> 
> Diff: https://reviews.apache.org/r/66041/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> David Mollitor
> 
>

Review Request 66041: [HIVE-18343] Remove LinkedList from ColumnStatsSemanticAnalyzer.java

2018-03-13 Thread David Mollitor 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66041/
---

Review request for hive.


Repository: hive-git


Description
---

[HIVE-18343] Remove LinkedList from ColumnStatsSemanticAnalyzer.java


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
c97e2a9 


Diff: https://reviews.apache.org/r/66041/diff/1/


Testing
---


Thanks,

David Mollitor

[jira] [Created] (HIVE-18950) DESCRIBE EXTENDED missing details of default constraint

2018-03-13 Thread Aswathy Chellammal Sreekumar (JIRA) 
Aswathy Chellammal Sreekumar created HIVE-18950:
---

 Summary: DESCRIBE EXTENDED missing details of default constraint
 Key: HIVE-18950
 URL: https://issues.apache.org/jira/browse/HIVE-18950
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aswathy Chellammal Sreekumar
Assignee: Vineet Garg
 Fix For: 3.0.0


Describe extended output is missing default constraint details
{noformat}
0: jdbc:hive2://ctr-e138-1518143905142-95188-> create table t1(j int constraint 
c1 default 4);
INFO  : Compiling 
command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318): 
create table t1(j int constraint c1 default 4)
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318); Time 
taken: 0.015 seconds
INFO  : Executing 
command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318): 
create table t1(j int constraint c1 default 4)
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318); Time 
taken: 0.048 seconds
INFO  : OK
No rows affected (0.087 seconds)
{noformat}


{noformat}
0: jdbc:hive2://ctr-e138-1518143905142-95188-> DESCRIBE EXTENDED t1;
INFO  : Compiling 
command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc): 
DESCRIBE EXTENDED t1
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, 
type:string, comment:from deserializer), FieldSchema(name:data_type, 
type:string, comment:from deserializer), FieldSchema(name:comment, type:string, 
comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc); Time 
taken: 0.029 seconds
INFO  : Executing 
command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc): 
DESCRIBE EXTENDED t1
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc); Time 
taken: 0.03 seconds
INFO  : OK
+-++--+
|  col_name   | data_type   
   | comment  |
+-++--+
| j   | int 
   |  |
| | NULL
   | NULL |
| Detailed Table Information  | Table(tableName:t1, dbName:default, 
owner:hrt_qa, createTime:1520972931, lastAccessTime:0, retention:0, 
sd:StorageDescriptor(cols:[FieldSchema(name:j, type:int, comment:null)], 
location:hdfs://mycluster/apps/hive/warehouse/t1, 
inputFormat:org.apache.hadoop.mapred.TextInputFormat, 
outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, 
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, 
serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, 
parameters:{serialization.format=1}), bucketCols:[], sortCols:[], 
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], 
skewedColValueLocationMaps:{}), storedAsSubDirectories:false), 
partitionKeys:[], parameters:{totalSize=0, numRows=0, rawDataSize=0, 
transactional_properties=insert_only, 
COLUMN_STATS_ACCURATE={\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"j\":\"true\"}},
 numFiles=0, transient_lastDdlTime=1520972931, transactional=true}, 
viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, 
rewriteEnabled:false) |  |
+-++--+
3 rows selected (0.099 seconds){noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199118
---




service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 666 (patched)


JIRA? this seems pretty critical


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199114
---




service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 663 (patched)


should this be addressed? the starting sessionstate will not be present iirc



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 670 (patched)


the error handling for SSL tests has disappeared



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 707 (patched)


there's a logic change here... WM should still be activated w/o resource 
plan if queue is present, so that activating a resource plan could enable it


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199113
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java
Line 55 (original), 56 (patched)


same as other comment - why change many places to a more general type just 
to add casts in other places?
at least before the dependency on specific type was compile time verified, 
now it's run time verified



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 183 (patched)


until? should there be a followup jira? :)



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 198 (patched)


nit: maybe the latch listener can be hidden in a non-static nested class... 
otherwise the interface implementation is visible to the users and these 
methods are a bit confusing.



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 202 (patched)


just wondering, how is this protected from races?
is there some grace period? should this be configurable/documented?
I.e. is it possible to check hasLeadership() just before losing it, but 
still publish this?



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 268 (patched)


nit: javadoc


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199112
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
Lines 123 (patched)


nit: why is a thread needed when we just join immediately?



itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
Lines 178 (patched)


nit: could it wait or loop on some condition for 3sec instead of sleeping? 
e.g. hs2Instances.size or leaders.size below



llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
Line 330 (original), 330 (patched)


hmm... why does getInstances now get one instance?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Lines 67 (patched)


why is this change necessary? shouldn't worker ID be enough?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Line 65 (original), 73 (patched)


nit: this is missing another 31



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 84 (patched)


hmm... if these are moved here, should they still be parameters to ctor? I 
saw GROUP being added to all the ctors calling super(), but given that they are 
taken from here might as well just hide them inside here



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 86 (original), 91 (patched)


comment is out of date



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 168 (patched)


hmm.. I'm not sure what this change is about.
ACL provider may be called for multiple paths that do not exist; some may 
be top level paths that should have full access, and some should be user level 
paths that should be restricted.
Generally there's a clear boundary - everything above certain dir is open, 
everything at or below is restricted, so only one path argument is necessary.
Why is this change needed?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 406 (original), 459 (patched)


why not final?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 482 (patched)


should null be logged? not sure when it would be valid for correct data to 
have null here, if there's no bug of some sort



llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java
Lines 27 (patched)


hmm



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Line 1365 (original), 1375 (patched)


hmm... this cast doesn't look so good. Why was field type changed? Maybe 
the cast should be at assignment


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>

[jira] [Created] (HIVE-18949) Improve CompactorMR logging

2018-03-13 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-18949:
-

 Summary: Improve CompactorMR logging
 Key: HIVE-18949
 URL: https://issues.apache.org/jira/browse/HIVE-18949
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Eugene Koifman


It currently doesn't log anything useful at INFO level or above from the 
running JOB.  Should probably at least log the set of files read and produced 
and perhaps details of each Split



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18948) Acquire locks before generating the valid transaction list

2018-03-13 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18948:
--

 Summary: Acquire locks before generating the valid transaction list
 Key: HIVE-18948
 URL: https://issues.apache.org/jira/browse/HIVE-18948
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez


HIVE-18825 moves the valid transaction list generation logic before query 
optimization. In order to support lock-based concurrency control correctly, the 
logic to acquire the locks has to be moved before query optimization too, and 
before the valid transaction list is generated.

This requires a bit of work/refactoring, since lock acquisition logic relies on 
read/write entities, and it is heavily dependent on the QueryPlan object too, 
e.g., it sets some properties in the file sink descriptors for the plan. 
Currently, all these data structures (except for read entities) are only 
available after query has been optimized, hence we will need to 1) generate 
some of this data structures before query optimization, e.g., write entities, 
and 2) create and propagate some of the properties so they are set during query 
optimization, e.g., those properties contained in the file sink descriptors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18947) Add integration tests for HoS + kerberos

2018-03-13 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-18947:
---

 Summary: Add integration tests for HoS + kerberos 
 Key: HIVE-18947
 URL: https://issues.apache.org/jira/browse/HIVE-18947
 Project: Hive
  Issue Type: Bug
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Would be nice to have some unit tests that cover the Kerberos integration in 
{{SparkClientImpl}}. Might be worth exploring adding delegation token, and 
impersonation coverage too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 13, 2018, 6:41 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Rebased patch


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/3/

Changes: https://reviews.apache.org/r/65498/diff/2-3/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-18946) Fix columnstats merge NPE

2018-03-13 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18946:
---

 Summary: Fix columnstats merge NPE
 Key: HIVE-18946
 URL: https://issues.apache.org/jira/browse/HIVE-18946
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich


after analyzing an empty table may lead to an NPE when inserting into it...

{code}
2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main] 
metastore.RetryingHMSHandler: java.lang.NullPointerException
at 
org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
at 
org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
at 
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
[...]
{code}

reproduce

{code}
set hive.stats.autogather=true;
set hive.explain.user=true;

drop table if exists testdeci2;

create table testdeci2(
id int,
amount decimal(10,3),
sales_tax decimal(10,3),
item string)
stored as orc location '/tmp/testdeci2'
TBLPROPERTIES ("transactional"="false")
;


analyze table testdeci2 compute statistics for columns;

insert into table testdeci2 
values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18945) Support "analyze table T"

2018-03-13 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18945:
---

 Summary: Support "analyze table T"
 Key: HIVE-18945
 URL: https://issues.apache.org/jira/browse/HIVE-18945
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich


I think it would be good to have it behave the same as 
{code}
analyze table T compute statistics for columns
{code}
this could help people who not yet know the different analyze commands ; to run 
the probably  most appropriate one



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18944) Groupping sets position is set incorrectly during DPP

2018-03-13 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18944:
---

 Summary: Groupping sets position is set incorrectly during DPP
 Key: HIVE-18944
 URL: https://issues.apache.org/jira/browse/HIVE-18944
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


groupingSetsPosition is set to -1 in case there are no grouping sets; however 
DPP calls the constructor with 0 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18943) Hive building with Hadoop 3.0.0 ERROR

2018-03-13 Thread gehaijiang (JIRA) 
gehaijiang created HIVE-18943:
-

 Summary: Hive building with Hadoop 3.0.0  ERROR
 Key: HIVE-18943
 URL: https://issues.apache.org/jira/browse/HIVE-18943
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2
Reporter: gehaijiang


hive  version:  2.3.2 

hadoop version: 3.0.0 

building error:  

 

[WARNING] COMPILATION WARNING :
[INFO] -
[WARNING] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:
 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
 uses or overrides a deprecated API.
[WARNING] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:
 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
 uses unchecked or unsafe operations.
[WARNING] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:
 Recompile with -Xlint:unchecked for details.
[INFO] 4 warnings
[INFO] -
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1088,29]
 constructor DistCpOptions in class org.apache.hadoop.tools.DistCpOptions 
cannot be applied to given types;
 required: org.apache.hadoop.tools.DistCpOptions.Builder
 found: java.util.List,org.apache.hadoop.fs.Path
 reason: actual and formal argument lists differ in length
[ERROR] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1089,12]
 cannot find symbol
 symbol: method setSyncFolder(boolean)
 location: variable options of type org.apache.hadoop.tools.DistCpOptions
[ERROR] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1090,12]
 cannot find symbol
 symbol: method setSkipCRC(boolean)
 location: variable options of type org.apache.hadoop.tools.DistCpOptions
[ERROR] 
/home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1091,12]
 cannot find symbol
 symbol: method preserve(org.apache.hadoop.tools.DistCpOptions.FileAttribute)
 location: variable options of type org.apache.hadoop.tools.DistCpOptions
[INFO] 4 errors
[INFO] -
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Hive ... SUCCESS [ 50.616 s]
[INFO] Hive Shims Common .. SUCCESS [07:24 min]
[INFO] Hive Shims 0.23  FAILURE [04:17 min]
[INFO] Hive Shims Scheduler ... SKIPPED
[INFO] Hive Shims . SKIPPED
[INFO] Hive Common  SKIPPED
[INFO] Hive Service RPC ... SKIPPED
[INFO] Hive Serde . SKIPPED
[INFO] Hive Metastore . SKIPPED
[INFO] Hive Vector-Code-Gen Utilities . SKIPPED
[INFO] Hive Llap Common ... SKIPPED
[INFO] Hive Llap Client ... SKIPPED
[INFO] Hive Llap Tez .. SKIPPED
[INFO] Spark Remote Client  SKIPPED
[INFO] Hive Query Language  SKIPPED
[INFO] Hive Llap Server ... SKIPPED
[INFO] Hive Service ... SKIPPED
[INFO] Hive Accumulo Handler .. SKIPPED
[INFO] Hive JDBC .. SKIPPED
[INFO] Hive Beeline ... SKIPPED
[INFO] Hive CLI ... SKIPPED
[INFO] Hive Contrib ... SKIPPED
[INFO] Hive Druid Handler . SKIPPED
[INFO] Hive HBase Handler . SKIPPED
[INFO] Hive JDBC Handler .. SKIPPED
[INFO] Hive HCatalog .. SKIPPED
[INFO] Hive HCatalog Core . SKIPPED