On Tue, Apr 12, 2016 at 11:22 AM, Gary Helmling <ghelml...@gmail.com> wrote:
> ... > > > Currently in the master branch, HMaster hosts 'system tables': e.g. > > hbase:meta. HMaster is doing more than just gardening the cluster, > > bootstrapping and keeping all up and serving healthy as in branch-1; in > > master branch, it is actually in the write path for the most critical > > system regions. > > > > > I think it's important to point out that this feature exists and is usable > in branch-1 as well, including in all 1.x releases. It just disabled by > default branch-1 and enabled by default in master. So this is really a > comparison of an existing, shipping, feature that does work, and is being > used vs. ongoing development work in master. > > I did not realize this facility was being used in branch-1 or even that it worked well enough to be deployed to production in branch-1. > > > > > Let me provoke discussion by making the statement that we should undo > > HMaster carrying any regions ever; that the HMaster function is work > enough > > for a single dedicated server and that it important enough that it cannot > > take a background role on a serving RegionServer (I could go back from > this > > position if evidence HMaster role could be backgrounded). Notions of a > > Master carrying system tables only are just not on given system tables > will > > be too big for a single server especially when hbase:meta is split (so we > > can scale). > > > If we really think that normal master housekeeping functions are work > enough that we shouldn't combine with region serving, then why do we think > that those will _not_ have to be scaled by splitting the metadata space > across multiple servers when we encounter meta-scaling issues that require > splitting meta to distribute it across multiple servers? Master meta functions may one day grow such that they are more than one server can manage. Chatting w/ folks who have run a system that is like hbase's (smile) that is run at scale, rather than split the master function, they took the time to make the master more efficient so they didn't have to distribute its duties. We can split hbase:meta and distribute it around the cluster if an hbase:meta region can be served like any other region in the system. > If we really want > to scale, then it seems like we need to tackle scaling the region metadata > in general across multiple active masters, in which case meta-on-master is > not really an argument either way. > > Distributing the metadata function amongst a cluster of masters has come up before. But before we go there, a single master that does metadata function only rather than metadata function AND fielding all metadata reads will be able to do more metadata ops if the hbase:meta reads are done elsewhere. Rough experiments with more to follow show that this should get us to our next scaling target, 1M regions on a cluster. > > > This simple distinction of HMaster and RegionServer roles is > > also what our users know and have gotten used to so needs to be a good > > reason to change it (We can still pursue the single binary that can do > > HMaster or HRegionServer role determined at runtime). > > > > The distinction in roles in HBase has long been used as a criticism of > HBase's operational complexity. I think we would be doing our users a > service by simplifying this and making it a detail they do not need to > worry about. If we can truly make this transparent to users and improve > operability at the same time, I think that would be the best outcome. > I could go this route of the floating master after we'd done some work. First lets figure out this current state of hbase master branch where we have an inbetweenie, a master that is sort-of-a-regionserver carrying only system tables and thereby getting in the way of our being able to scale metadata function. St.Ack
