Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks who helped in getting this feature in this level.
Starting with my +1 (binding). # Tested a 5 node cluster with resource profiles enabled/disabled (feature is disabled by default) # All apis added are marked as Unstable/Evolving (very few) # There is no compatibility break with older versions (we have added UT cases also to ensure same) # Performance tests were done using SLS and also with some tight loops unit tests. There is no much regression with current trunk. # Latest jenkins +1 on YARN-7013 for whole branch code. # Verified old RM UI and new YARN UI (newly added resources could be seen easily) Once again thanks all the folks who helped in getting this feature. Kudos! Thanks - Sunil On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan <wheele...@gmail.com> wrote: > Hi folks, > > Per earlier discussion [1], I'd like to start a formal vote to merge > feature branch YARN-3926 (Resource profile) to trunk. The vote will run for > 7 days and will end August 30 10:00 AM PDT. > > Briefly, YARN-3926 can extend resource model of YARN to support resource > types other than CPU and memory, so it will be a cornerstone of features > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In > addition to that, YARN-3926 allows admin to preconfigure resource profiles > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB > disk>, so applications can request "m3.large" profile instead of specifying > all resource types’s values. > > There are 32 subtasks that were completed as part of this effort. > > This feature needs to be explicitly turned on before use. We paid close > attention to compatibility, performance, and scalability of this feature, > mentioned in [1], we didn't see observable performance regression in large > scale SLS (scheduler load simulator) executions and saw less than 5% > performance regression by using micro benchmark added by YARN-6775. > > This feature works from end-to-end (including UI/CLI/application/server), > we have setup a cluster with this feature turned on runs for several weeks, > we didn't see any issues by far. > > Merge JIRA: YARN-7013 (Jenkins gave +1 already). > Documentation: YARN-7056 > > Special thanks to a team of folks who worked hard and contributed towards > this effort including design discussion/development/reviews, etc.: Varun > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu, > Karthik Kambatla, Jason Lowe, Arun Suresh. > > Regards, > Wangda Tan > > [1] > > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%3DBbp4G8inQZmaMg%40mail.gmail.com%3E >
