JD, thanks for the jira and further explanation. for some reason, I was always thinking about 'pull' while considering the solution. Certainly 'push' the natural way to address this on Apache's hbase. I got your points now. Appreciate it... Demai
On Fri, Jul 26, 2013 at 1:08 PM, Jean-Daniel Cryans <[email protected]>wrote: > https://issues.apache.org/jira/browse/HBASE-9047 > > On Fri, Jul 26, 2013 at 12:59 PM, Jean-Daniel Cryans > <[email protected]> wrote: > > I guess I didn't explain my ideas clearly. > > > > So, first, replication in HBase is master-push, so you don't want to > > reverse the process. It means that this tool needs to run on the > > master cluster. > > > > Then I don't think you need to specify a timestamp since the > > replication state is in ZK. Basically that tool we're talking about > > would be able to read the replication state of each master region > > server, finish replicating what's missing, and then clear that state > > in zookeeper. > > > > The code that handles replication does most of that already. Check > > ReplicationSourceManager and ReplicationSource. Basically when > > ReplicationSourceManager.init() is called, it will check all the > > queues in ZK and try to grab those that aren't attached to a region > > server. If the whole cluster is down, it will grab all of them. > > > > The beautiful thing here is that you could start that tool on all your > > machines and the load will be spread out, but that might not be a big > > concern if replication wasn't lagging since it would take a few > > seconds to finish replicating the missing data for each region server. > > > > I'll open a jira. > > > > J-D > > > > On Fri, Jul 26, 2013 at 11:50 AM, Demai Ni <[email protected]> wrote: > >> JD, > >> > >> yeah. that sounds what I will need to do. a tool like this > >> [slave_cluster]$tool_to_syncup master_ZKquorum table_name > start_timestamp > >> > >> so two tasks for me: > >> 1) identify the start_timestamp > >> 2) write the tool_to_syncup which will reach to master_ZK, copy the > HLOGs > >> from makster, replay the HLOGs on Slave. > >> > >> are you aware of some example code for the 2) task that I can leverage? > >> thanks > >> > >> Demai >
