[
https://issues.apache.org/jira/browse/TWILL-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terence Yim resolved TWILL-87.
------------------------------
Resolution: Fixed
Merged and pushed to master.
> Adding Container Placement control (Placement Policy API)
> ----------------------------------------------------------
>
> Key: TWILL-87
> URL: https://issues.apache.org/jira/browse/TWILL-87
> Project: Apache Twill
> Issue Type: New Feature
> Components: api, yarn
> Affects Versions: 0.3.0-incubating
> Environment: Tested on Hadoop Yarn 2.2 and 2.3 running on
> Ubuntu-nodes (4 GB , 8 Cores) cluster.
> Reporter: Gourav Khaneja
> Assignee: Gourav Khaneja
> Labels: features, github
> Fix For: 0.4.0-incubating
>
> Attachments:
> 0001-TWILL-87-Adding-Container-Placement-Policy-control.patch,
> 0001-TWILL-87-Adding-Container-Placement-Policy-control.patch
>
>
> Yarn AMRMClient provides API to control container placement. We need to
> enhance Twill API so that user can specify container placement policies.
> Twill could use AMRMClient to try allocating containers according to
> specified placement policy.
> Added Placement Policy API in TwillSpecification. For now, the Placement
> policy type includes: (a) DISTRIBUTED, which tries to spawn specified
> runnables on different hosts. (b) DEFAULT, i.e. no special placement policy
> requirements.
> Implementation Detail: The DISTRIBUTED runnable instances are provisioned one
> after another (as opposed to grouping provision requests in one allocate call
> based on ResourceSpecs). AM blacklists the hosts on which existing
> DISTRIBUTED runnables are running. If no container is provisioned for
> MAX_CONSTRAINED_PROVISION_ATTEMPTS number of attempts, AM relaxes blacklist
> constraints (or any other constraint).
> Also, it make sense to specify Hosts and Racks through Placement Policy API
> instead of using Resource Specification. So, moved that logic into Placement
> Policy too.
>
> Tests
> 1. Added unit tests to test Placement Policy (using MiniYarnCluster)
> (a) Specify DISTRIBUTED runnables, runnables with Hosts and Racks
> in a Twill App and verify all constraints and appropriately honored.
> (b) Specify DISTRIBUTED and DEFAULT runnables in a Twill App and
> verify all constraints and appropriately honored. Increase number of
> instances for all runnables and verify all constraints and appropriately
> honored.
> (c) Tested DISTRIBUTED placement policy under stress (i.e. not
> enough resources available to honor constraints). Verify that AM relaxes
> constraints and try it's best.
> 2. Tested on a real Cluster.
>
> Please review the API and changes in PR -
> https://github.com/apache/incubator-twill/pull/7
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
