[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-06-17 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050983#comment-13050983
 ] 

Hudson commented on HBASE-1502:
---

Integrated in HBase-TRUNK #1976 (See 
[https://builds.apache.org/job/HBase-TRUNK/1976/])


> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, 
> 1502-v7.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-05-06 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030227#comment-13030227
 ] 

Hudson commented on HBASE-1502:
---

Integrated in HBase-TRUNK #1909 (See 
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])


> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, 
> 1502-v7.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-29 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027261#comment-13027261
 ] 

Hudson commented on HBASE-1502:
---

Integrated in HBase-TRUNK #1888 (See 
[https://builds.apache.org/hudson/job/HBase-TRUNK/1888/])


> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, 
> 1502-v7.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-27 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025915#comment-13025915
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--



bq.  On 2011-04-27 00:17:30, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/ClusterStatus.java, line 113
bq.  > 
bq.  >
bq.  > Can just use this.liveServers.values() for here and below?

value is HServerLoad (Not a List of HRegionInfo).  We want count of regions.

I did remove the dup iteration having getAverageLoad instead call 
getRegionsCount.


bq.  On 2011-04-27 00:17:30, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HServerInfo.java, line 42
bq.  > 
bq.  >
bq.  > i see webuiport below, does this TODO still apply?

It does.

HSI is deprecated and after this patch goes in, there is no means of the RS 
telling Master of its webui port -- in master we'll rely on configs.  In rare 
case where webui port changes -- it can if occupied -- then master will be off.

Will address elsewhere.  Will write json to the RS ephemeral node with 
'metadata' about the RS that will include webui port but likely other stuff 
that could be factored doing loading calculations, etc. (it used to hold a HSA 
but thats been bumped in this patch).  Will file an issue after this patch goes 
in.


bq.  On 2011-04-27 00:17:30, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java, line 83
bq.  > 
bq.  >
bq.  > why String and not ServerName?  because master has no startcode?  (i 
see use of ServerName for master above tho)

I changed it (This method is not used anywhere seemingly).


bq.  On 2011-04-27 00:17:30, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HServerAddress.java, line 42
bq.  > 
bq.  >
bq.  > Where is this actually used now?  Should point it out here so it's 
clear and so that when it goes away we know we can get rid of this.

Will do.  Had a change of heart last night; previous was going to just 
'sort-of' deprecate HSA but on way  home, after thinking on it, HSA is just 
never a good idea so will outright deprecate it.


bq.  On 2011-04-27 00:17:30, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 792
bq.  > 
bq.  >
bq.  > this is because HSA actually makes a connection or does the lookup?

This I'll purge.  I'll use the getConnection(ISA) that this patch adds instead 
so no reference to HSA in HBaseAdmin (except deprecated uses)


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/#review568
---


On 2011-04-26 23:50:23, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/647/
bq.  ---
bq.  
bq.  (Updated 2011-04-26 23:50:23)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch does not completely remove heartbeats.  It unburdens the 
heartbeat of control messages; now heartbeat is used to
bq.  send the master load only (At most recent hackathon we had rough agreement 
that we'd keep heartbeat to carry load)... if we miss some, no biggie.   
bq.  
bq.  RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
bq.  We pass a String now instead of HServerAddress so this should help with 
our DNS issues where the two sides disagree.
bq.  
bq.  Removed HMsg.
bq.  
bq.  HServerAddress as been sort_of_deprecated.  Its in our API so can't remove 
it easily (its embedded inside HRegionLocation).
bq.  Otherwise, we don't use it internally anymore.
bq.  
bq.  HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.
bq.  
bq.  Fixed up regionserver and master startup so they now look the same.
bq.  
bq.  New tests
bq.  
bq.  Cruft cleanup.
bq.  
bq.  
bq.  This addresses bug hbase-1502.
bq.  https://issues.apache.org/jira/browse/hbase-1502
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 5701639 
bq.src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
bq.src/main/java/org/apa

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-27 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025914#comment-13025914
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/#review585
---


Thanks for review jg


src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java


This I've been changing removing HSA.  Instead I'm passing an ISA (this 
patch adds a getConnection(isa) method).


- Michael


On 2011-04-26 23:50:23, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/647/
bq.  ---
bq.  
bq.  (Updated 2011-04-26 23:50:23)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch does not completely remove heartbeats.  It unburdens the 
heartbeat of control messages; now heartbeat is used to
bq.  send the master load only (At most recent hackathon we had rough agreement 
that we'd keep heartbeat to carry load)... if we miss some, no biggie.   
bq.  
bq.  RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
bq.  We pass a String now instead of HServerAddress so this should help with 
our DNS issues where the two sides disagree.
bq.  
bq.  Removed HMsg.
bq.  
bq.  HServerAddress as been sort_of_deprecated.  Its in our API so can't remove 
it easily (its embedded inside HRegionLocation).
bq.  Otherwise, we don't use it internally anymore.
bq.  
bq.  HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.
bq.  
bq.  Fixed up regionserver and master startup so they now look the same.
bq.  
bq.  New tests
bq.  
bq.  Cruft cleanup.
bq.  
bq.  
bq.  This addresses bug hbase-1502.
bq.  https://issues.apache.org/jira/browse/hbase-1502
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 5701639 
bq.src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
bq.src/main/java/org/apache/hadoop/hbase/HRegionLocation.java bd353b8 
bq.src/main/java/org/apache/hadoop/hbase/HServerAddress.java 7f8a472 
bq.src/main/java/org/apache/hadoop/hbase/HServerInfo.java 0b5bd94 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 2372053 
bq.src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 0d696ab 
bq.src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1da9742 
bq.src/main/java/org/apache/hadoop/hbase/Server.java df396fa 
bq.src/main/java/org/apache/hadoop/hbase/ServerName.java PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/avro/AvroUtil.java d7a1e67 
bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 
be31179 
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java c2ee031 
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 6e22cf5 
bq.src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java 
aee64c5 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java afb666a 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 2bb4725 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
c348f7a 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java edacf56 
bq.
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
 6c62024 
bq.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 
8df6aa4 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 
d64817f 
bq.src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java c22e342 
bq.src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java 
a55f9d6 
bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 
d8f8463 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java ec28de4 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
25139b3 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 663cab5 
bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 2273e55 
bq.src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java 
66a3345 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
58fdb30 
bq.src/main/java/org/apache/hadoop/hbase/master/DeadServer.java 05600c4 
bq.src/main/java/org/apache/h

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-26 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025526#comment-13025526
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/#review568
---

Ship it!


woohoo!  glad HMsg is dead!


src/main/java/org/apache/hadoop/hbase/ClusterStatus.java


Can just use this.liveServers.values() for here and below?



src/main/java/org/apache/hadoop/hbase/HServerAddress.java


Where is this actually used now?  Should point it out here so it's clear 
and so that when it goes away we know we can get rid of this.



src/main/java/org/apache/hadoop/hbase/HServerInfo.java


i see webuiport below, does this TODO still apply?



src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java


why String and not ServerName?  because master has no startcode?  (i see 
use of ServerName for master above tho)



src/main/java/org/apache/hadoop/hbase/ServerName.java


awesome that this is tucked away in here now



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java


this is because HSA actually makes a connection or does the lookup?


- Jonathan


On 2011-04-26 23:50:23, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/647/
bq.  ---
bq.  
bq.  (Updated 2011-04-26 23:50:23)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch does not completely remove heartbeats.  It unburdens the 
heartbeat of control messages; now heartbeat is used to
bq.  send the master load only (At most recent hackathon we had rough agreement 
that we'd keep heartbeat to carry load)... if we miss some, no biggie.   
bq.  
bq.  RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
bq.  We pass a String now instead of HServerAddress so this should help with 
our DNS issues where the two sides disagree.
bq.  
bq.  Removed HMsg.
bq.  
bq.  HServerAddress as been sort_of_deprecated.  Its in our API so can't remove 
it easily (its embedded inside HRegionLocation).
bq.  Otherwise, we don't use it internally anymore.
bq.  
bq.  HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.
bq.  
bq.  Fixed up regionserver and master startup so they now look the same.
bq.  
bq.  New tests
bq.  
bq.  Cruft cleanup.
bq.  
bq.  
bq.  This addresses bug hbase-1502.
bq.  https://issues.apache.org/jira/browse/hbase-1502
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 5701639 
bq.src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
bq.src/main/java/org/apache/hadoop/hbase/HRegionLocation.java bd353b8 
bq.src/main/java/org/apache/hadoop/hbase/HServerAddress.java 7f8a472 
bq.src/main/java/org/apache/hadoop/hbase/HServerInfo.java 0b5bd94 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 2372053 
bq.src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 0d696ab 
bq.src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1da9742 
bq.src/main/java/org/apache/hadoop/hbase/Server.java df396fa 
bq.src/main/java/org/apache/hadoop/hbase/ServerName.java PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/avro/AvroUtil.java d7a1e67 
bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 
be31179 
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java c2ee031 
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 6e22cf5 
bq.src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java 
aee64c5 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java afb666a 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 2bb4725 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
c348f7a 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java edacf56 
bq.
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
 6c62024 
bq.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 
8df6aa4 
b

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-26 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025520#comment-13025520
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/
---

(Updated 2011-04-26 23:50:23.656187)


Review request for hbase.


Changes
---

All tests pass now.  I'd like to get this patch in soon.  I'm currently 
spending a good bit of my time trying to keep this patch up with current TRUNK. 
 I'd rather commit and then address issues after.

This version of the patch does one  make significant change though in that it 
deprecates prewarmRegionCache.  IMO this is a burdensome feature that is little 
used; i'd like to have it die off.


Summary
---

This patch does not completely remove heartbeats.  It unburdens the heartbeat 
of control messages; now heartbeat is used to
send the master load only (At most recent hackathon we had rough agreement that 
we'd keep heartbeat to carry load)... if we miss some, no biggie.   

RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
We pass a String now instead of HServerAddress so this should help with our DNS 
issues where the two sides disagree.

Removed HMsg.

HServerAddress as been sort_of_deprecated.  Its in our API so can't remove it 
easily (its embedded inside HRegionLocation).
Otherwise, we don't use it internally anymore.

HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.

Fixed up regionserver and master startup so they now look the same.

New tests

Cruft cleanup.


This addresses bug hbase-1502.
https://issues.apache.org/jira/browse/hbase-1502


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
  src/main/java/org/apache/hadoop/hbase/HConstants.java 5701639 
  src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
  src/main/java/org/apache/hadoop/hbase/HRegionLocation.java bd353b8 
  src/main/java/org/apache/hadoop/hbase/HServerAddress.java 7f8a472 
  src/main/java/org/apache/hadoop/hbase/HServerInfo.java 0b5bd94 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 2372053 
  src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 0d696ab 
  src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1da9742 
  src/main/java/org/apache/hadoop/hbase/Server.java df396fa 
  src/main/java/org/apache/hadoop/hbase/ServerName.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/avro/AvroUtil.java d7a1e67 
  src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java be31179 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java c2ee031 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 6e22cf5 
  src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java aee64c5 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java afb666a 
  src/main/java/org/apache/hadoop/hbase/client/HConnection.java 2bb4725 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java c348f7a 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java edacf56 
  
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
 6c62024 
  src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 
8df6aa4 
  src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java d64817f 
  src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java c22e342 
  src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java 
a55f9d6 
  src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d8f8463 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java ec28de4 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 25139b3 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 663cab5 
  src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 2273e55 
  src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java 66a3345 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 58fdb30 
  src/main/java/org/apache/hadoop/hbase/master/DeadServer.java 05600c4 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 79a48ba 
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java 6c92cbc 
  src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
4bb072e 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 55e0162 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 04befe9 
  src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java dada818 
  
src/main/java/org/apache/hadoop/hbase/master/handler/MetaS

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-24 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024693#comment-13024693
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/
---

(Updated 2011-04-24 23:44:37.199842)


Review request for hbase.


Changes
---

Brought this up to the tip of TRUNK.  Most tests pass.  Looking into a 
distributed split failure.


Summary
---

This patch does not completely remove heartbeats.  It unburdens the heartbeat 
of control messages; now heartbeat is used to
send the master load only (At most recent hackathon we had rough agreement that 
we'd keep heartbeat to carry load)... if we miss some, no biggie.   

RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
We pass a String now instead of HServerAddress so this should help with our DNS 
issues where the two sides disagree.

Removed HMsg.

HServerAddress as been sort_of_deprecated.  Its in our API so can't remove it 
easily (its embedded inside HRegionLocation).
Otherwise, we don't use it internally anymore.

HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.

Fixed up regionserver and master startup so they now look the same.

New tests

Cruft cleanup.


This addresses bug hbase-1502.
https://issues.apache.org/jira/browse/hbase-1502


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
  src/main/java/org/apache/hadoop/hbase/HConstants.java 5701639 
  src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
  src/main/java/org/apache/hadoop/hbase/HRegionLocation.java bd353b8 
  src/main/java/org/apache/hadoop/hbase/HServerAddress.java 7f8a472 
  src/main/java/org/apache/hadoop/hbase/HServerInfo.java 0b5bd94 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 2372053 
  src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 0d696ab 
  src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1da9742 
  src/main/java/org/apache/hadoop/hbase/Server.java df396fa 
  src/main/java/org/apache/hadoop/hbase/ServerName.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/avro/AvroUtil.java d7a1e67 
  src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java be31179 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java c2ee031 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 6e22cf5 
  src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java aee64c5 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java afb666a 
  src/main/java/org/apache/hadoop/hbase/client/HConnection.java 2bb4725 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java c348f7a 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java edacf56 
  
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
 6c62024 
  src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 
8df6aa4 
  src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java d64817f 
  src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java c22e342 
  src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java 
a55f9d6 
  src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d8f8463 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java ec28de4 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 25139b3 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 663cab5 
  src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java 66a3345 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 58fdb30 
  src/main/java/org/apache/hadoop/hbase/master/DeadServer.java 05600c4 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 79a48ba 
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java 6c92cbc 
  src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
4bb072e 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 55e0162 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 04befe9 
  
src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
 eb01a6a 
  src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 
c478ab7 
  
src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java 
60662c2 
  src/main/java/org/apache/hadoop/hbase/master/handler/SplitRegionHandler.java 
fee0b3d 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c065702 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-23 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023608#comment-13023608
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/
---

(Updated 2011-04-23 21:51:08.451039)


Review request for hbase.


Changes
---

This diff goes against tip of TRUNK


Summary
---

This patch does not completely remove heartbeats.  It unburdens the heartbeat 
of control messages; now heartbeat is used to
send the master load only (At most recent hackathon we had rough agreement that 
we'd keep heartbeat to carry load)... if we miss some, no biggie.   

RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
We pass a String now instead of HServerAddress so this should help with our DNS 
issues where the two sides disagree.

Removed HMsg.

HServerAddress as been sort_of_deprecated.  Its in our API so can't remove it 
easily (its embedded inside HRegionLocation).
Otherwise, we don't use it internally anymore.

HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.

Fixed up regionserver and master startup so they now look the same.

New tests

Cruft cleanup.


This addresses bug hbase-1502.
https://issues.apache.org/jira/browse/hbase-1502


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
  src/main/java/org/apache/hadoop/hbase/HConstants.java 5701639 
  src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
  src/main/java/org/apache/hadoop/hbase/HRegionLocation.java bd353b8 
  src/main/java/org/apache/hadoop/hbase/HServerAddress.java 7f8a472 
  src/main/java/org/apache/hadoop/hbase/HServerInfo.java 0b5bd94 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 2372053 
  src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 0d696ab 
  src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1da9742 
  src/main/java/org/apache/hadoop/hbase/Server.java df396fa 
  src/main/java/org/apache/hadoop/hbase/ServerName.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/avro/AvroUtil.java d7a1e67 
  src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java be31179 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java c2ee031 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 6e22cf5 
  src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java aee64c5 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java afb666a 
  src/main/java/org/apache/hadoop/hbase/client/HConnection.java 2bb4725 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java c348f7a 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java edacf56 
  
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
 6c62024 
  src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 
8df6aa4 
  src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java d64817f 
  src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java c22e342 
  src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java 
a55f9d6 
  src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d8f8463 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java ec28de4 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 25139b3 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 663cab5 
  src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java 66a3345 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 58fdb30 
  src/main/java/org/apache/hadoop/hbase/master/DeadServer.java 05600c4 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 79a48ba 
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java 6c92cbc 
  src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
4bb072e 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 55e0162 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 04befe9 
  
src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
 eb01a6a 
  src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 
c478ab7 
  
src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java 
60662c2 
  src/main/java/org/apache/hadoop/hbase/master/handler/SplitRegionHandler.java 
fee0b3d 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c065702 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java d0a1e11 
  src/main/java/org/apache/hadoop/hbase/regionserver/

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-22 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023465#comment-13023465
 ] 

jirapos...@reviews.apache.org commented on HBASE-1502:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/647/
---

Review request for hbase.


Summary
---

This patch does not completely remove heartbeats.  It unburdens the heartbeat 
of control messages; now heartbeat is used to
send the master load only (At most recent hackathon we had rough agreement that 
we'd keep heartbeat to carry load)... if we miss some, no biggie.   

RPC version changed on HMasterRegionInfo since the regionServerStartup and 
regionServerReport arguments have changed.
We pass a String now instead of HServerAddress so this should help with our DNS 
issues where the two sides disagree.

Removed HMsg.

HServerAddress as been sort_of_deprecated.  Its in our API so can't remove it 
easily (its embedded inside HRegionLocation).
Otherwise, we don't use it internally anymore.

HServerInfo is deprecated.  Server meta data is now available in new class 
ServerName and load lives apart from HSI now.

Fixed up regionserver and master startup so they now look the same.

New tests

Cruft cleanup.


This addresses bug hbase-1502.
https://issues.apache.org/jira/browse/hbase-1502


Diffs
-

  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 26a8bef 
  src/main/java/org/apache/hadoop/hbase/HConstants.java ce0ea12 
  src/main/java/org/apache/hadoop/hbase/HMsg.java 87beb00 
  src/main/java/org/apache/hadoop/hbase/HRegionLocation.java bd353b8 
  src/main/java/org/apache/hadoop/hbase/HServerAddress.java 7f8a472 
  src/main/java/org/apache/hadoop/hbase/HServerInfo.java 7aa8bd9 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 2372053 
  src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 0d696ab 
  src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1da9742 
  src/main/java/org/apache/hadoop/hbase/Server.java df396fa 
  src/main/java/org/apache/hadoop/hbase/ServerName.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/avro/AvroUtil.java d7a1e67 
  src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java be31179 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java c2ee031 
  src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 6e22cf5 
  src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java aee64c5 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java e022fe4 
  src/main/java/org/apache/hadoop/hbase/client/HConnection.java d8a2fc3 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 53decd6 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java bb3a8fa 
  
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java
 6c62024 
  src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 
9576c48 
  src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java db0870b 
  src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java de13e27 
  src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java 
a55f9d6 
  src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d8f8463 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java ec28de4 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 25139b3 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 663cab5 
  src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java 66a3345 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java e9b2af2 
  src/main/java/org/apache/hadoop/hbase/master/DeadServer.java efcbb99 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 62789d3 
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java fe5f7e1 
  src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
c0e9417 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 7b5181a 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 4b8a59a 
  
src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
 eb01a6a 
  src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 
c478ab7 
  
src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java 
60662c2 
  src/main/java/org/apache/hadoop/hbase/master/handler/SplitRegionHandler.java 
fee0b3d 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 516d06d 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 85b4144 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
4051e0b 
  src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactio

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-04-22 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023463#comment-13023463
 ] 

stack commented on HBASE-1502:
--

Added a patch that is ready for review to https://reviews.apache.org/r/647/  
I'm still testing but its ready for input.

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, 
> 1502-v7.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase

2011-03-21 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009368#comment-13009368
 ] 

stack commented on HBASE-1502:
--

The Master needs to know cluster load for load balancing and so we can display 
load aggregate in the UI.  Currently this is carried on the back of the 
heatbeat as a serialized HServerLoad.  The intent was to put this loading 
elsewhere than up on the heartbeat.  Thought was to put it out in zk.

At the hackathon, it was thought that on a cluster of any reasonable size, the 
number of writes would be too high (Think 700 node cluster writing load every 
1-5 seconds).  Todd raised the issue after talking with Patrick; zk blocks 
reads while taking on writes so our read rate would go down.

Chatting more, the thought was to just leave up the heartbeat connection -- 
call it something else though (jgray) -- but remove HMsg and just pass HSLs.   
We talked of master querying on a period all RSs but for now, we have something 
that works so just keep it up it was thought.

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, 
> 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2011-03-03 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002423#comment-13002423
 ] 

Hudson commented on HBASE-1502:
---

Integrated in HBase-TRUNK #1767 (See 
[https://hudson.apache.org/hudson/job/HBase-TRUNK/1767/])


> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, 
> 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2011-02-14 Thread Jonathan Gray (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994394#comment-12994394
 ] 

Jonathan Gray commented on HBASE-1502:
--

Got it.  That sounds like a good start and removing the heartbeat/HMsg is far 
more critical than the startup RPC.  I remember some issue in the past where it 
was odd that discovery was RPC but failure was ZK... but don't recall what 
exactly it was now.

+1 on ZK content being JSON serialized.

Just to bring it up since it's loosely related to this stuff, I'm of the 
opinion that a second monitor process is eventually going to be necessary.  ZK 
timeouts are just too high and there are many cases where if we could take GC 
pauses out of the equation, we could have much faster failure detection.  A 
second process that would not have any GC issues could have an ephemeral node 
with a much smaller timeout, or it could monitor the RS process and system 
itself.  I don't see another way towards reducing failure detection times 
without triggering false-positives when the RS is in a GC pause (a 
"recoverable" fault).

On an even more unrelated note, we could have some kind of metric (or this 
other process could figure out) how often GC pauses are happening / for how 
long (either through a looping sleep() thread or an RPC to the process) and use 
that as an additional balancing metric.  Or we could have it so once it passes 
a threshold, we shed the regions off of that RS (actually flushing instead of 
needing replay), and then restart the RS process.

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-v2.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2011-02-14 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994380#comment-12994380
 ] 

stack commented on HBASE-1502:
--

Yeah, 'registration' is via an RPC.  The RS sends over startcode and port only 
(should probably pass the webui port too).  The master returns the hostname 
only.  On receipt, the RS writes up into ZK, a znode whose name is the 
'servername -- i.e. the master passed domain, port, and startcode -- and whose 
content is the RS 'load' updated every 5 minutes or so.  I think load should be 
JSON serialized (I think all zk content should be JSON serialized)

I'm willing to entertain suggestions on how we might do away with the startup 
RPC but we need to get to the RS the hostname the Master sees; currently it 
does this by doing lookup on the remote ip the RPCServer sees on registration 
RPC.  The RS needs what the master  has for a hostname so when it writes 
.META., it writes the same name the master will write.

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-v2.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2011-02-13 Thread Jonathan Gray (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994195#comment-12994195
 ] 

Jonathan Gray commented on HBASE-1502:
--

Only did cursory look at patch, but looks like this still has it so startup is 
via RPC but shutdown (will eventually be) via ZK... What exactly will be 
transmitted each way in that startup RPC?  Is it necessary?

Also, how does master get server load info now?

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-v2.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2011-02-13 Thread Jonathan Gray (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994196#comment-12994196
 ] 

Jonathan Gray commented on HBASE-1502:
--

(yay HMsg is gone!)

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Task
>Reporter: Nitay Joffe
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1502-v2.txt, 1502.txt
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2010-12-07 Thread Jonathan Gray (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969083#action_12969083
 ] 

Jonathan Gray commented on HBASE-1502:
--

We're also using the regionServerStartup or regionServerReport to determine an 
RS is online (we use the ZK node to determine when it dies).  This stuff would 
also need some cleanup/rework to completely drop heartbeats.

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Wish
>Reporter: Nitay Joffe
> Fix For: 0.92.0
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2010-12-07 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968980#action_12968980
 ] 

stack commented on HBASE-1502:
--

We are almost there.  If we could get rid of the split message and shutdown 
being passed by HMsg, we'd be able to drop HMsg and just rely completely on zk 
client ping.

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Wish
>Reporter: Nitay Joffe
> Fix For: 0.92.0
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2010-09-01 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905133#action_12905133
 ] 

stack commented on HBASE-1502:
--

Post hbase-2692, heartbeats:

+ Carry to the master the HServerInfo from each regionserver.
+ When a split, a message from rs tells master about the it; master uses info 
to update its inmemory map of cluster
+ Master tells rs to shutdown (needed currently if rs is carrying a meta region)

I was thinking that RSs could publish load info to zk on a period -- every 
minute or 5 (some fraction of the balancer interval).

For split info, need to flag master somehow and needs to be mechanism where the 
split can be rediscovered if missed for some reason (The anti-entropy doohickey 
where clients tell master of 'problems' would be one such way).

On shutdown message, I need to do a little more smarts around shutdown and that 
should eliminate the shutdown message (need to expose shutdown as something a 
client can invoke against an RS).


> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Wish
>Reporter: Nitay Joffe
> Fix For: 0.92.0
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-1502) Remove need for heartbeats in HBase

2010-08-31 Thread Jonathan Gray (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904924#action_12904924
 ] 

Jonathan Gray commented on HBASE-1502:
--

Following HBASE-2692, I think the only thing we're really using the heartbeat 
for is the mostLoaded servers.  But we're actually tracking all assignment 
in-memory anyways, so maybe we can do this now?  Shutdown stuff could come on 
it's own RPC?

> Remove need for heartbeats in HBase
> ---
>
> Key: HBASE-1502
> URL: https://issues.apache.org/jira/browse/HBASE-1502
> Project: HBase
>  Issue Type: Wish
>Reporter: Nitay Joffe
> Fix For: 0.92.0
>
>
> HBase currently uses heartbeats between region servers and the master, 
> piggybacking information on them when it can. This issue is to investigate if 
> we can get rid of the need for those using ZooKeeper events.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.