[
https://issues.apache.org/jira/browse/YARN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449183#comment-15449183
]
Lei Guo commented on YARN-3926:
-------------------------------
[~vvasudev], some thoughts I just commented in Yarn-4793.
As this Jira targets to build a unified interface, I'd like to share some
thoughts related to the resource modeling part. The core of Yarn is still to
map the resource and workload. For the resource modeling proposed in YARN-3926,
it extends the current Yarn static resource modeling to be a flat resource
modeling. The end user has the potential to define/schedule their own resource.
I am considering whether we should do further extension to make the resource
modeling to be a hierarchy based modeling. The use case I see for future is the
heterogenous environment with different hardware accelerators (GPU, Intel Xeon
Phi, FPGA, etc). For example, if you treat one GPU as a unit of special
resource, the flat resource modeling is good enough. But we are seeing cases
that GPU to be shared between applications, even the application prefer to
allocate certain range of memory inside GPU to avoid cache rotation issue. In
this case, it's hard for scheduler to handle. There is relationship between
resource (just like the relationship between applications in Slider). Scheduler
must allocate GPU memory and GPU core on the same GPU.
If we do have vision to cover more complicate environments with Yarn, maybe
it's time to consider further extension on the resource modeling together with
Slider integration and unified service API.
> Extend the YARN resource model for easier resource-type management and
> profiles
> -------------------------------------------------------------------------------
>
> Key: YARN-3926
> URL: https://issues.apache.org/jira/browse/YARN-3926
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, resourcemanager
> Reporter: Varun Vasudev
> Assignee: Varun Vasudev
> Attachments: Proposal for modifying resource model and profiles.pdf
>
>
> Currently, there are efforts to add support for various resource-types such
> as disk(YARN-2139), network(YARN-2140), and HDFS bandwidth(YARN-2681). These
> efforts all aim to add support for a new resource type and are fairly
> involved efforts. In addition, once support is added, it becomes harder for
> users to specify the resources they need. All existing jobs have to be
> modified, or have to use the minimum allocation.
> This ticket is a proposal to extend the YARN resource model to a more
> flexible model which makes it easier to support additional resource-types. It
> also considers the related aspect of “resource profiles” which allow users to
> easily specify the various resources they need for any given container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
