Hi All, Inline with earlier communication dated 17th July 2018, I would like to provide some updates.
We are approaching previously proposed code freeze date (Aug 31). One of the critical feature Node Attributes feature merge discussion/vote is ongoing. Also few other Blocker bugs need a bit more time. With regard to this, suggesting to push the feature/code freeze for 2 more weeks to accommodate these jiras too. Proposing Updated changes in plan inline with this: Feature freeze date : all features to merge by September 7, 2018. Code freeze date : blockers/critical only, no improvements and blocker/critical bug-fixes September 14, 2018. Release date: September 28, 2018 If any features in branch which are targeted to 3.2.0, please reply to this email thread. *Here's an updated 3.2.0 feature status:* 1. Merged & Completed features: - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads Initial cut. - (Uma) HDFS-10285: HDFS Storage Policy Satisfier - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler. - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API and CLI. 2. Features close to finish: - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote Ongoing. - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2. Patch in progress. - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works. - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In progress. 3. Tentative features: - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be done before Aug 2018. - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging as more discussions are on-going. *Summary of 3.2.0 issues status:* 26 Blocker and Critical issues [1] are open, I am following up with owners to get status on each of them to get in by Code Freeze date. [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER BY priority DESC Thanks, Sunil On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sun...@apache.org> wrote: > Hi All, > > Thanks for the feedbacks. Inline with earlier communication dated 17th > July 2018, I would like to provide some updates. > > We are approaching previously proposed feature freeze date (Aug 21, about > 7 days from today). > If any features in branch which are targeted to 3.2.0, please reply to > this email thread. > Steve has mentioned about the s3 features which will come close to Code > Freeze Date (Aug 31st). > > *Here's an updated 3.2.0 feature status:* > > 1. Merged & Completed features: > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads > Initial cut. > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier > > 2. Features close to finish: > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches > are all in, only one last > patch is in review state. > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler. > Close to commit. > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API > and CLI. 2 patches are pending > which will be closed by Feature freeze date. > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2. > Patch in progress. > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works. > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. > In progress. > > 3. Tentative features: > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be > done before Aug 2018. > - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging > as more discussions are on-going. > > *Summary of 3.2.0 issues status:* > > 39 Blocker and Critical issues [1] are open, I am checking with owners to > get status on each of them to get in by Code Freeze date. > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER > BY priority DESC > > Thanks, > Sunil > > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sun...@apache.org> wrote: > >> Thanks Subru for the thoughts. >> One of the main reason for a major release is to push out critical >> features with a faster cadence to the users. If we are pulling more and >> more different types of features to a minor release, that branch will >> become more destabilized and it may be tough to say that 3.1.2 is stable >> that 3.1.1 for eg. We always tend to improve and stabilize features in >> subsequent minor release. >> For few companies, it makes sense to push out these new features faster >> to make a reach to the users. Adding to the point to the backporting >> issues, I agree that its a pain and we can workaround that with some git >> scripts. If we can make such scripts available to committers, backport will >> be seem-less across branches and we can achieve the faster release cadence >> also. >> >> Thoughts? >> >> - Sunil >> >> >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote: >> >>> Thanks Sunil for volunteering to lead the release effort. I am generally >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we >>> already >>> have too many branches to be maintained. I already see many commits are >>> in >>> different branches with no apparent rationale, for e.g: 3.1 has commits >>> which are absent in 3.0 etc. >>> >>> Additionally AFAIK 3.x has not been deployed in any major production >>> setting so the cost of adding features should be minimal. >>> >>> Thoughts? >>> >>> -Subru >>> >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sun...@apache.org> wrote: >>> >>> > Thanks Steve, Aaron, Wangda for sharing thoughts. >>> > >>> > Yes, important changes and features are much needed, hence we will be >>> > keeping the door open for them as possible. Also considering few more >>> > offline requests from other folks, I think extending the timeframe by >>> > couple of weeks makes sense (including a second RC buffer) and this >>> should >>> > ideally help us to ship this by September itself. >>> > >>> > Revised dates (I will be updating same in Roadmap wiki as well) >>> > >>> > - Feature freeze date : all features to merge by August 21, 2018. >>> > >>> > - Code freeze date : blockers/critical only, no improvements and non >>> > blocker/critical >>> > >>> > bug-fixes August 31, 2018. >>> > >>> > - Release date: September 15, 2018 >>> > >>> > Thank Eric and Zian, I think Wangda has already answered your >>> questions. >>> > >>> > Thanks >>> > Sunil >>> > >>> > >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheele...@gmail.com> >>> wrote: >>> > >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that. >>> > > >>> > > To concerns from Steve, >>> > > >>> > > It is a good idea to keep the door open to get important changes / >>> > > features in before cutoff. I would prefer to keep the proposed >>> release >>> > date >>> > > to make sure things can happen earlier instead of last minute and we >>> all >>> > > know that releases are always get delayed :). I'm also fine if we >>> want >>> > get >>> > > another several weeks time. >>> > > >>> > > Regarding of 3.3 release, I would suggest doing that before >>> thanksgiving. >>> > > Do you think is it good or too early / late? >>> > > >>> > > Eric, >>> > > >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get >>> merged >>> > > in time, we probably not need the YARN-8220. >>> > > >>> > > Sunil, >>> > > >>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/ >>> > Roadmap >>> > > with proposed plan as well? We can fill feature list first before >>> getting >>> > > consensus of time. >>> > > >>> > > Thanks, >>> > > Wangda >>> > > >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri >>> <fab...@cloudera.com.invalid >>> > > >>> > > wrote: >>> > > >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran < >>> ste...@hortonworks.com> >>> > >> wrote: >>> > >> >>> > >> > >>> > >> > >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sun...@apache.org<mailto: >>> > >> > sun...@apache.org>> wrote: >>> > >> > >>> > >> > I would also would like to take this opportunity to come up with a >>> > >> detailed >>> > >> > plan. >>> > >> > >>> > >> > - Feature freeze date : all features should be merged by August >>> 10, >>> > >> 2018. >>> > >> > >>> > >> > >>> > >> > >>> > >> > <snip> >>> > >> >>> > >> > >>> > >> > Please let me know if I missed any features targeted to 3.2 per >>> this >>> > >> > >>> > >> > >>> > >> > Well there these big todo lists for S3 & S3Guard. >>> > >> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226 >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220 >>> > >> > >>> > >> > >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2 >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407 >>> > >> > >>> > >> > I don't think this is quite ready yet, I've been doing work on >>> it, but >>> > >> if >>> > >> > we have a 3 week deadline, I'm going to expect some timely >>> reviews on >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546 >>> > >> > >>> > >> > I've uprated that to a blocker feature; will review the S3 & >>> S3Guard >>> > >> JIRAs >>> > >> > to see which of those are blocking. Then there are some pressing >>> > "guave, >>> > >> > java 9 prep" >>> > >> > >>> > >> > >>> > >> I can help with this part if you like. >>> > >> >>> > >> >>> > >> >>> > >> > >>> > >> > >>> > >> > >>> > >> > timeline. I would like to volunteer myself as release manager of >>> 3.2.0 >>> > >> > release. >>> > >> > >>> > >> > >>> > >> > well volunteered! >>> > >> > >>> > >> > >>> > >> > >>> > >> Yes, thank you for stepping up. >>> > >> >>> > >> >>> > >> > >>> > >> > I think this raises a good q: what timetable should we have for >>> the >>> > >> 3.2. & >>> > >> > 3.3 releases; if we do want a faster cadence, then having the >>> outline >>> > >> time >>> > >> > from the 3.2 to the 3.3 release means that there's less concern >>> about >>> > >> > things not making the 3.2 dealine >>> > >> > >>> > >> > -Steve >>> > >> > >>> > >> > >>> > >> Good idea to mitigate the short deadline. >>> > >> >>> > >> -AF >>> > >> >>> > > >>> > >>> >>
