Hi Daniel Thank you very much for the support.
* When you say that the feature can be turned off, do you mean resource types or resource profiles? I know there's an off-by-default property that governs resource profiles, but I didn't see any way to turn off resource types. Yes,*yarn.resourcemanager.resource-profiles.enabled* is false by default and controls off/on of this feature. Now regarding new resource types, its been loaded from "*resource-types.xml"* and by default this XML file is not available in the package. Thus prevents any issues in default case. Once this file is added to a cluster then new resources will be loaded from same. * Even if only CPU and memory are configured, i.e. no additional resource types, the code path is different than it was. Earlier primitive data types were used to represent vcores and memory. As per resource profile work, all resources under YARN is categorized as ResourceInformation and placed under existing Resource object. So memory and vcores will be accessible and operable with same set of public apis from Resources or ResourceCalculator (DRC) same as earlier even when feature is off (Code path is same, but improved to support a unified ResourceInformation class instead of memory/vcores primitive types). Thanks Sunil On Sat, Aug 26, 2017 at 8:10 PM Daniel Templeton <dan...@cloudera.com> wrote: > Quick question, Wangda. When you say that the feature can be turned > off, do you mean resource types or resource profiles? I know there's an > off-by-default property that governs resource profiles, but I didn't see > any way to turn off resource types. Even if only CPU and memory are > configured, i.e. no additional resource types, the code path is > different than it was. Specifically, where CPU and memory were > primitives before, they're now entries in an array whose indexes have to > be looked up through the ResourceUtils class. Did I miss something? > > For those who haven't followed the feature closely, there are really two > features here. Resource types allows for declarative extension of the > resource system in YARN. Resource profiles builds on top of resource > types to allow a user to request a group of resources as a profile, much > like EC2 instance types, e.g. "fast-compute" might mean 32GB RAM, 8 > vcores, and 2 GPUs. > > Daniel > > On 8/23/17 11:49 AM, Wangda Tan wrote: > > Hi folks, > > > > Per earlier discussion [1], I'd like to start a formal vote to merge > > feature branch YARN-3926 (Resource profile) to trunk. The vote will run > for > > 7 days and will end August 30 10:00 AM PDT. > > > > Briefly, YARN-3926 can extend resource model of YARN to support resource > > types other than CPU and memory, so it will be a cornerstone of features > > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA > > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In > > addition to that, YARN-3926 allows admin to preconfigure resource > profiles > > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB > > disk>, so applications can request "m3.large" profile instead of > specifying > > all resource types’s values. > > > > There are 32 subtasks that were completed as part of this effort. > > > > This feature needs to be explicitly turned on before use. We paid close > > attention to compatibility, performance, and scalability of this feature, > > mentioned in [1], we didn't see observable performance regression in > large > > scale SLS (scheduler load simulator) executions and saw less than 5% > > performance regression by using micro benchmark added by YARN-6775. > > > > This feature works from end-to-end (including UI/CLI/application/server), > > we have setup a cluster with this feature turned on runs for several > weeks, > > we didn't see any issues by far. > > > > Merge JIRA: YARN-7013 (Jenkins gave +1 already). > > Documentation: YARN-7056 > > > > Special thanks to a team of folks who worked hard and contributed towards > > this effort including design discussion/development/reviews, etc.: Varun > > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu, > > Karthik Kambatla, Jason Lowe, Arun Suresh. > > > > Regards, > > Wangda Tan > > > > [1] > > > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%3DBbp4G8inQZmaMg%40mail.gmail.com%3E > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
