[
https://issues.apache.org/jira/browse/YARN-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241900#comment-15241900
]
Wangda Tan commented on YARN-4902:
----------------------------------
[~asuresh],
Thanks for explanations,
bq. Establishment of an application/affinity/server group should be part of a
deployment schema API separate from the ResourceRequest.
Does k8s have separate resource request APIs? It seems k8s combines resource
request and launch context to the same [spec
scheme|http://kubernetes.io/docs/user-guide/pods/multi-container/#the-spec-schema].
My understanding of deployment scheme should be very close to
ContainerLaunchContext. (after we decouple container and allocation, it could
become AllocationDeploymentContext).
IAW, we already have deployment API in YARN. And placement-related fields
should be a part of resource request API.
bq. My fear is that coupling allocation-tags and target-allocation-tags with
the resource request would lead to a far more complex implementation than
necessary. For ex, One must now enforce an order in which ResourceRequests are
made to ensure that applications/components/containers specified in
target-allocation-tags exist / infact already been deployed.
I'm not sure if I understand your proposal correctly, I cannot find it
simplifies the problem:
For example, in your solution. If applications want anti-affinity between
applications (application_1 and application_2 don't want their allocations on
the same host). Applications needs to get the "group-id" first, to get the
"group-id", it will send a request to get a new "anti-affinity-group-id", and
the anti-affinity-group-id" will be passed to application-1 and application-2,
and app1/app2 submitted the group-id to scheduler while requesting resources.
If we don't have the group-id, which is solution described in design doc, app_1
/ app_2 can set a pre-generated id as a part of allocation_tag, and make
requests indicate (*not generated-allocation-tag*).
It seems to me that complexity of two solutions are similar, and the group-id
solution needs a separate scheduler API.
bq. Also having groups creates prior to the resource request is issued allows
for easier/faster determination of whether a request is satisfiable in the
first place.
In my mind affinity / anti-affinity should be late-binding. For example, if an
app wants to get resources close to "hbase_region_server", if there's no
running / pending hbase_region_server allocation. The app should wait till such
allocations available.
> [Umbrella] Generalized and unified scheduling-strategies in YARN
> ----------------------------------------------------------------
>
> Key: YARN-4902
> URL: https://issues.apache.org/jira/browse/YARN-4902
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Wangda Tan
> Attachments: Generalized and unified scheduling-strategies in YARN
> -v0.pdf
>
>
> Apache Hadoop YARN's ResourceRequest mechanism is the core part of the YARN's
> scheduling API for applications to use. The ResourceRequest mechanism is a
> powerful API for applications (specifically ApplicationMasters) to indicate
> to YARN what size of containers are needed, and where in the cluster etc.
> However a host of new feature requirements are making the API increasingly
> more and more complex and difficult to understand by users and making it very
> complicated to implement within the code-base.
> This JIRA aims to generalize and unify all such scheduling-strategies in YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
