+1 (binding) Thanks Konstantin for leading the merge effort!
I worked very closely with Chen, Konstantin, and Erik in the testing stage and I feel confident that the feature has now completed designed functionalities and has proven to be stable. Great team work with contributors from multiple companies! On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko <shv.had...@gmail.com> wrote: > Hi Hadoop developers, > > I would like to propose to merge to trunk the feature branch HDFS-12943 for > Consistent Reads from Standby Node. The feature is intended to scale read > RPC workloads. On large clusters reads comprise 95% of all RPCs to the > NameNode. We should be able to accommodate higher overall RPC workloads (up > to 4x by some estimates) by adding multiple ObserverNodes. > > The main functionality has been implemented see sub-tasks of HDFS-12943. > We followed up with the test plan. Testing was done on two independent > clusters (see HDFS-14058 and HDFS-14059) with security enabled. > We ran standard HDFS commands, MR jobs, admin commands including manual > failover. > We know of one cluster running this feature in production. > > There are a few outstanding issues: > 1. Need to provide proper documentation - a user guide for the new feature > 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't > know about ObserverNodes trying to convert them to SBNs. > 3. Scale testing and performance fine-tuning > 4. As testing progresses, we continue fixing non-critical bugs like > HDFS-14116. > > I attached a unified patch to the umbrella jira for the review and Jenkins > build. > Please vote on this thread. The vote will run for 7 days until Wed Dec 12. > > Thanks, > --Konstantin > -- Zhe Zhang Apache Hadoop Committer http://zhe-thoughts.github.io/about/ | @oldcap
