subject:"Re\\\\\\\: Review Request 65498\\\\\\\: HIVE\\\\\\\-18281\\\\\\\: HiveServer2 HA for LLAP and Workload Manager"

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-15 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 15, 2018, 7:03 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

minor update to avoid zk exceptions during shutdown. also unregister listener 
is removed as latch close will automatically remove listeners and no need to do 
explicit removal.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 907d064 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/6/

Changes: https://reviews.apache.org/r/65498/diff/5-6/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-14 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 15, 2018, 5:51 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

some changes to notification handling. added comments about how notifications 
are received and handled.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 907d064 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/5/

Changes: https://reviews.apache.org/r/65498/diff/4-5/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-14 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 14, 2018, 8:24 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Addressed review comments.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 30c24d9 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/4/

Changes: https://reviews.apache.org/r/65498/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j



> On March 13, 2018, 8:41 p.m., Sergey Shelukhin wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
> > Lines 123 (patched)
> > 
> >
> > nit: why is a thread needed when we just join immediately?

good catch.. join() supposed to happen together for the different threads.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199112
---


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j



> On March 13, 2018, 9:03 p.m., Sergey Shelukhin wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java
> > Lines 663 (patched)
> > 
> >
> > should this be addressed? the starting sessionstate will not be present 
> > iirc

Fixed.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199114
---


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199139
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java
Line 55 (original), 56 (patched)


Moved the casts internally to getInstances()



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 198 (patched)


Fixed.



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 202 (patched)


Good point. It is possible to lose leadership just before publishing it. 
Added a lock to protect around races. also reordered to move hasLeadership 
happen just before publishing endpoint.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 666 (patched)


HIVE-18952 created



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 670 (patched)


Sorry bad rebase. Fixed it.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 707 (patched)


Fixed it.


- Prasanth_J


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199136
---




llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
Line 330 (original), 330 (patched)


Fixed method signature. A path can have only one instance.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Lines 67 (patched)


UUID is static. This is for the unit test.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 84 (patched)


these are defaults. llap and tez AM uses the defaults but HS2 will override 
or ignore some prefixes.. need c'tor override for all prefixes.

LLAP path:
/llap-sasl/hiveuser/hostname/workers/worker-000

HS2 path:
/hs2ActivePassiveHA-sasl/instances/instance-00



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 168 (patched)


This is required for ZK based service discovery. Clients connecting using 
ZK based service discovery to HS2 will have namespace in the URL (cannot have 
subpaths like /ns/user/workers). 

JDBC URL will have hs2ActivePassiveHA as namespace.

For HS2 the ACL provider path will be
/hs2ActivePassiveHA-sasl/instances/

There is no per user restriction (JDBC URL doesn't get users).



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 406 (original), 459 (patched)


IntelliJ rule violation :) final is unnecessary for static methods.



llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java
Lines 27 (patched)


llap-common is not dependent on llap-client. Dropped..



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Line 1365 (original), 1375 (patched)


ServiceRegistry interface had LlapServiceInstanceSet which is specific to 
LLAP. ServiceRegistry is updated to return generic ServiceInstanceSet. 
HiveServer2HAInstanceSet extends it to add isLeader() interface on top of 
ServiceRegistry.


- Prasanth_J


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199118
---




service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 666 (patched)


JIRA? this seems pretty critical


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199114
---




service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 663 (patched)


should this be addressed? the starting sessionstate will not be present iirc



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 670 (patched)


the error handling for SSL tests has disappeared



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 707 (patched)


there's a logic change here... WM should still be activated w/o resource 
plan if queue is present, so that activating a resource plan could enable it


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199113
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java
Line 55 (original), 56 (patched)


same as other comment - why change many places to a more general type just 
to add casts in other places?
at least before the dependency on specific type was compile time verified, 
now it's run time verified



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 183 (patched)


until? should there be a followup jira? :)



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 198 (patched)


nit: maybe the latch listener can be hidden in a non-static nested class... 
otherwise the interface implementation is visible to the users and these 
methods are a bit confusing.



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 202 (patched)


just wondering, how is this protected from races?
is there some grace period? should this be configurable/documented?
I.e. is it possible to check hasLeadership() just before losing it, but 
still publish this?



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 268 (patched)


nit: javadoc


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199112
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
Lines 123 (patched)


nit: why is a thread needed when we just join immediately?



itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
Lines 178 (patched)


nit: could it wait or loop on some condition for 3sec instead of sleeping? 
e.g. hs2Instances.size or leaders.size below



llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
Line 330 (original), 330 (patched)


hmm... why does getInstances now get one instance?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Lines 67 (patched)


why is this change necessary? shouldn't worker ID be enough?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Line 65 (original), 73 (patched)


nit: this is missing another 31



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 84 (patched)


hmm... if these are moved here, should they still be parameters to ctor? I 
saw GROUP being added to all the ctors calling super(), but given that they are 
taken from here might as well just hide them inside here



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 86 (original), 91 (patched)


comment is out of date



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 168 (patched)


hmm.. I'm not sure what this change is about.
ACL provider may be called for multiple paths that do not exist; some may 
be top level paths that should have full access, and some should be user level 
paths that should be restricted.
Generally there's a clear boundary - everything above certain dir is open, 
everything at or below is restricted, so only one path argument is necessary.
Why is this change needed?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 406 (original), 459 (patched)


why not final?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 482 (patched)


should null be logged? not sure when it would be valid for correct data to 
have null here, if there's no bug of some sort



llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java
Lines 27 (patched)


hmm



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Line 1365 (original), 1375 (patched)


hmm... this cast doesn't look so good. Why was field type changed? Maybe 
the cast should be at assignment


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 13, 2018, 6:41 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Rebased patch


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/3/

Changes: https://reviews.apache.org/r/65498/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-02-05 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated Feb. 5, 2018, 11:05 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Added /peers endpoint to list all hs2 instances and their leadership status


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 99e8457 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
82fdf6c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java b33f027 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 2a528cd 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/2/

Changes: https://reviews.apache.org/r/65498/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

13 matches

Site Navigation

Mail list logo

Footer information