GitHub user bbayani opened a pull request:
https://github.com/apache/flink/pull/4628
[FLINK-7486]:[flink-mesos]:Support for adding unique attribute / grouâ¦
â¦p_by attribute constraints
JIRA issue: FLINK-7486
## What is the purpose of the change
- To be able to add soft constraint to balance the tasks across
mesos-agents on basis of host constraint
## Brief change log
- Added new config 'mesos.constraints.soft.balanced'
- Parsed the config in MesosTaskManagerParams and prepared list of
BalancedConstraintParam objects
- Prepared Set of TaskIds from workersInNew and workersInLaunch in
MesosResourceManager/MesosFlinkResourceManager. Used this Set in coTaskGetter
to get co-task-ids
- Updated LaunchableMesosWorker to return soft constraints from
BalancedConstraintParam objects
## Verifying this change
- Manually verified the change by running flink on mesos cluster. The
mesos worker had attributes AZ and hostname set. After adding the constraint,
taskmanagers were launched on mesos-workers with unique AZ/hostname values.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: don't know
- The runtime per-record code paths (performance sensitive): don't know
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes (flink deployment on
mesos)
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
The feature is controlled by config 'mesos.constraints.soft.balanced'
in flink-conf.yaml
The config is documented in docs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bbayani/flink soft_balanced_constraint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4628.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4628
----
commit c333344ba0b57d94bb3e1ae7c88e50d2a8549a8b
Author: bbayani <[email protected]>
Date: 2017-08-30T16:36:45Z
[FLINK-7486]:[flink-mesos]:Support for adding unique attribute / group_by
attribute constraints
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
