[ https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-18846: -------------------------- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2 and master. Thanks for the review [~mdrob] > Accommodate the hbase-indexer/lily/SEP consumer deploy-type > ----------------------------------------------------------- > > Key: HBASE-18846 > URL: https://issues.apache.org/jira/browse/HBASE-18846 > Project: HBase > Issue Type: Bug > Reporter: stack > Assignee: stack > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18846.master.001.patch, > HBASE-18846.master.002.patch, HBASE-18846.master.003.patch, > HBASE-18846.master.004.patch, HBASE-18846.master.005.patch, > HBASE-18846.master.006.patch, HBASE-18846.master.007.patch, > HBASE-18846.master.007.patch, IndexerConnection.java, hbase-site.xml, > javadoc.txt > > > This is a follow-on from HBASE-10504, Define a Replication Interface. There > we defined a new, flexible replication endpoint for others to implement but > it did little to help the case of the lily hbase-indexer. This issue takes up > the case of the hbase-indexer. > The hbase-indexer poses to hbase as a 'fake' peer cluster (For why > hbase-indexer is implemented so, the advantage to having the indexing done in > a separate process set that can be independently scaled, can participate in > the same security realm, etc., see discussion in HBASE-10504). The > hbase-indexer will start up a cut-down "RegionServer" processes that are just > an instance of hbase RpcServer hosting an AdminProtos Service. They make > themselves 'appear' to the Replication Source by hoisting up an ephemeral > znode 'registering' as a RegionServer. The source cluster then streams > WALEdits to the Admin Protos method: > {code} > public ReplicateWALEntryResponse replicateWALEntry(final RpcController > controller, > final ReplicateWALEntryRequest request) throws ServiceException { > {code} > The hbase-indexer relies on other hbase internals like Server so it can get a > ZooKeeperWatcher instance and know the 'name' to use for this cut-down server. > Thoughts on how to proceed include: > > * Better formalize its current digestion of hbase internals; make it so > rpcserver is allowed to be used by others, etc. This would be hard to do > given they use basics like Server, Protobuf serdes for WAL types, and > AdminProtos Service. Any change in this wide API breaks (again) > hbase-indexer. We have made a 'channel' for Coprocessor Endpoints so they > continue to work though they use 'internal' types. They can use protos in > hbase-protocol. hbase-protocol protos are in a limbo currently where they are > sort-of 'public'; a TODO. Perhaps the hbase-indexer could do similar relying > on the hbase-protocol (pb2.5) content and we could do something to reveal > rpcserver and zk for hbase-indexer safe use. > * Start an actual RegionServer only have it register the AdminProtos Service > only -- not ClientProtos and the Service that does Master interaction, etc. > [I checked, this is not as easy to do as I at first thought -- St.Ack] Then > have the hbase-indexer implement an AdminCoprocessor to override the > replicateWALEntry method (the Admin CP implementation may need work). This > would narrow the hbase-indexer exposure to that of the Admin Coprocessor > Interface > * Over in HBASE-10504, [~enis] suggested "... if we want to provide > isolation for the replication services in hbase, we can have a simple host as > another daemon which hosts the ReplicationEndpoint implementation. RS's will > use a built-in RE to send the edits to this layer, and the host will delegate > it to the RE implementation. The flow would be something like: RS --> RE > inside RS --> Host daemon for RE --> Actual RE implementation --> third party > system..." > > Other crazy notions occur including the setup of an Admin Interface > Coprocessor Endpoint. A new ReplicationEndpoint would feed the replication > stream to the remote cluster via the CPEP registered channel. > But time is short. Hopefully we can figure something that will work in 2.0 > timeframe w/o too much code movement. -- This message was sent by Atlassian JIRA (v6.4.14#64029)