Really looking forward to getting this in. Couple of questions: * Can you maybe comment a bit on the type of scale testing done ? Specifically, the number of resources tested with and any point where it is discovered that performance might take a hit. Also, given that we do not have AM's that currently use this feature, can you folks point me to any test application / framework or has this been integrated with MapReduce ? * Is there a plan to merge this with branch-2 ? - Since we would like to see this in 2.9.0 as well.
Just to clarify, I am a +1 for merging, irrespective of the above - given that this is an opt-in feature after all. I am just eager to start using it :) Cheers -Arun On Thu, Aug 24, 2017 at 10:54 AM, Sunil G <sun...@apache.org> wrote: > Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks who > helped in getting this feature in this level. > > Starting with my +1 (binding). > > > # Tested a 5 node cluster with resource profiles enabled/disabled (feature > is disabled by default) > > # All apis added are marked as Unstable/Evolving (very few) > > # There is no compatibility break with older versions (we have added UT > cases also to ensure same) > > # Performance tests were done using SLS and also with some tight loops unit > tests. There is no much regression with current trunk. > > # Latest jenkins +1 on YARN-7013 for whole branch code. > > # Verified old RM UI and new YARN UI (newly added resources could be seen > easily) > > > Once again thanks all the folks who helped in getting this feature. Kudos! > > > Thanks > > - Sunil > > > On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan <wheele...@gmail.com> wrote: > > > Hi folks, > > > > Per earlier discussion [1], I'd like to start a formal vote to merge > > feature branch YARN-3926 (Resource profile) to trunk. The vote will run > for > > 7 days and will end August 30 10:00 AM PDT. > > > > Briefly, YARN-3926 can extend resource model of YARN to support resource > > types other than CPU and memory, so it will be a cornerstone of features > > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA > > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In > > addition to that, YARN-3926 allows admin to preconfigure resource > profiles > > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB > > disk>, so applications can request "m3.large" profile instead of > specifying > > all resource types’s values. > > > > There are 32 subtasks that were completed as part of this effort. > > > > This feature needs to be explicitly turned on before use. We paid close > > attention to compatibility, performance, and scalability of this feature, > > mentioned in [1], we didn't see observable performance regression in > large > > scale SLS (scheduler load simulator) executions and saw less than 5% > > performance regression by using micro benchmark added by YARN-6775. > > > > This feature works from end-to-end (including UI/CLI/application/server), > > we have setup a cluster with this feature turned on runs for several > weeks, > > we didn't see any issues by far. > > > > Merge JIRA: YARN-7013 (Jenkins gave +1 already). > > Documentation: YARN-7056 > > > > Special thanks to a team of folks who worked hard and contributed towards > > this effort including design discussion/development/reviews, etc.: Varun > > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu, > > Karthik Kambatla, Jason Lowe, Arun Suresh. > > > > Regards, > > Wangda Tan > > > > [1] > > > > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/ > 201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4% > 3DBbp4G8inQZmaMg%40mail.gmail.com%3E > > >
