https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates
2017-10-20 Apologies for skipping the update last week. Here's how we're tracking for GA. Highlights: - Merge of HDFS router-based federation and API-based scheduler configuration with no reported problems. Kudos to the contributors involved! Red flags: - We're making a last-minute push to get resource types (but not resource profiles in). Coming this late, it's a risk, but we decided it's worthwhile for this feature. See Daniel's yarn-dev email <https://lists.apache.org/thread.html/cc51a68f1cf21c641cf77383ee1a13e3b2ca93d2a56f5a441f42b21b@%3Cyarn-dev.hadoop.apache.org%3E> for the full rationale. - Still uncovering EC bugs from testing Previously tracked GA blockers that have been resolved or dropped: - YARN-6623 <https://issues.apache.org/jira/browse/YARN-6623?src=confmacro> - Add support to turn off launching privileged containers in the container-executor RESOLVED: Committed and resolved - Change of ExecutionType - YARN-7275 <https://issues.apache.org/jira/browse/YARN-7275?src=confmacro> - NM Statestore cleanup for Container updates RESOLVED : Patch committed, resolved. - ReservationSystem - YARN-4859 <https://issues.apache.org/jira/browse/YARN-4859?src=confmacro> - [Bug] Unable to submit a job to a reservation when using FairScheduler RESOLVED: Yufei tested this and found things mostly worked, filed two not-blocker followons: YARN-7347 <https://issues.apache.org/jira/browse/YARN-7347?src=confmacro> - Fixe the bug in Fair scheduler to handle a queue named "root.root" OPEN and YARN-7348 <https://issues.apache.org/jira/browse/YARN-7348?src=confmacro> - Ignore the vcore in reservation request for fair policy queue OPEN GA blockers: - Change of ExecutionType - YARN-7178 <https://issues.apache.org/jira/browse/YARN-7178?src=confmacro> - Add documentation for Container Update API OPEN : Still no update from Arun, I pinged it. - ReservationSystem - YARN-4827 <https://issues.apache.org/jira/browse/YARN-4827?src=confmacro> - Document configuration of ReservationSystem for FairScheduler OPEN: Yufei said he'd work on it as of 2 days ago - Rolling upgrade - YARN-6142 <https://issues.apache.org/jira/browse/YARN-6142?src=confmacro> - Support rolling upgrade between 2.x and 3.x OPEN : I pinged this and asked for a status update - HDFS-11096 <https://issues.apache.org/jira/browse/HDFS-11096?src=confmacro> - Support rolling upgrade between 2.x and 3.xPATCH AVAILABLE: I pinged this and asked for a status update - Erasure coding - HDFS-12682 <https://issues.apache.org/jira/browse/HDFS-12682?src=confmacro> - ECAdmin -listPolicies will always show policy state as DISABLED OPEN: New blocker filed this week, Xiao is working on it - HDFS-12686 <https://issues.apache.org/jira/browse/HDFS-12686?src=confmacro> - Erasure coding system policy state is not correctly saved and loaded during real cluster restart OPEN: New blocker filed this week, Sammi is on it - HDFS-12686 <https://issues.apache.org/jira/browse/HDFS-12686?src=confmacro> - Erasure coding system policy state is not correctly saved and loaded during real cluster restart OPEN: Old blocker, Huafeng is on it, waiting on review from Wei-Chiu or Sammi Features merged for GA: - Erasure coding - Continued bug reporting and fixing based on testing at Cloudera. - Two new blockers filed this week, mentioned above. - Huafeng completed patch to reenable disabled EC tests - Classpath isolation (HADOOP-11656) - HADOOP-13916 <https://issues.apache.org/jira/browse/HADOOP-13916?src=confmacro> - Document how downstream clients should make use of the new shaded client artifacts IN PROGRESS: I pinged it - Compat guide (HADOOP-13714 <https://issues.apache.org/jira/browse/HADOOP-13714>) - HADOOP-14876 <https://issues.apache.org/jira/browse/HADOOP-14876?src=confmacro> - Create downstream developer docs from the compatibility guidelines PATCH AVAILABLE: Daniel has a patch up, revved based on Steve's review feedback, waiting on Steve's reply - HADOOP-14875 <https://issues.apache.org/jira/browse/HADOOP-14875?src=confmacro> - Create end user documentation from the compatibility guidelines OPEN: No patch yet - TSv2 alpha 2 - This was merged, no problems thus far [image: (smile)] - API-based scheduler configuration YARN-5734 <https://issues.apache.org/jira/browse/YARN-5734?src=confmacro> - OrgQueue for easy CapacityScheduler queue configuration management RESOLVED - Merged, no problems thus far [image: (smile)] - HDFS router-based configuration HDFS-10467 <https://issues.apache.org/jira/browse/HDFS-10467?src=confmacro> - Router-based HDFS federation RESOLVED - Merged, no problems thus far [image: (smile)] Unmerged features: - Resource types / profiles (YARN-3926 <https://issues.apache.org/jira/browse/YARN-3926> and YARN-7069 <https://issues.apache.org/jira/browse/YARN-7069>) (Wangda Tan) - We're going to try and get in resource types for 3.0.0 and leave resource profiles for 3.1.0. Daniel is spearheading this and other major contributors like Wangda and Sunil are onboard with the plan. Branch has been created and undergone testing, I expect a merge vote ASAP. - This is our biggest remaining risk. - YARN native services (YARN-5079 <https://issues.apache.org/jira/browse/YARN-5079>) (Jian He) - YARN-7351 <https://issues.apache.org/jira/browse/YARN-7351?src=confmacro> - High CPU usage issue in RegistryDNS PATCH AVAILABLE is blocking - Allen is reviewing YARN-7127 with some design-level questions about the API and architecture - This doesn't look like this is getting in based on YARN-7127 discussion