> > # Without meta on master, we double assign and lose data. > > I doubt meta on master solve this problem. > This has more to do on the fact that balancer, assignment, split, merge > are disjoint operations that are not aware of each other. > also those operation in general consist of multiple steps and if the master > crashes you may end up in an inconsistent state. > > Meta-on-master does dramatically improve things. For example, it makes it possible to cold-start HBase under load, where a non-meta-serving master is never able to successfully complete initialization. This is the difference between a cluster being able to come to a healthy state vs. one that is never able to complete assignments, communicate those assignments to clients and come to a steady state.
> this is what proc-v2 should solve. since we are aware of each operation > there is no chance of double assignment and similar by design. > > Again, I think it is difficult to compare an existing feature that is working in production use vs. one that is actively being developed in master. Preventing double assignment sounds great. When happens when the update of meta to communicate this to clients fails? So long as meta is served elsewhere you still have distributed state. Until we have an alternative that is feature complete and has demonstrated success and stability in production use, I don't see how we can even propose removing a feature that is solving real problems. I also think that this proposed direction will amplify our release problems and get us further away from regular, incremental releases. Master will remain unreleaseable indefinitely until proc v2 development is finished, and even initial releases will have problems that need to be ironed out. Ironing out issues in initial releases is not unexpected, but by removing existing solution we would be forcing a big-bang approach where everything has to work before anyone can move over to 2.0, which increases pressure for users to stay on 1.x releases, which increases pressure to backport features and brings us closer to the Hadoop way. I would much rather see us working on incrementally improving what we have and proving out new solutions piece by piece.
