[
https://issues.apache.org/jira/browse/FLINK-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369900#comment-16369900
]
ASF GitHub Bot commented on FLINK-8711:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/5532
[FLINK-8711] [yarn] Remove code which auto-magically changes slots per TM
## What is the purpose of the change
The FlinkYarnSessionCli auto-magically amends the number of slots with
which the
TMs are started. This was thought as a convenience function. However, it
breaks the
execution of jobs where we have multiple slot sharing groups. Therefore,
this commit
removes this code.
As a consequence, the user has to make sure that he specifies enough slots
and
containers for the pre Flip-6 code such that the number of total slots is
greater
or equal than the parallelism of the job.
**Important:** This PR changes Flink's behaviour wrt starting Flink Yarn
session clusters. It can become necessary that the user specifies the correct
number of slots with which he wants to launch his TMs.
## Verifying this change
This change is already covered by existing tests, such as the Yarn test
cases.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes it changes Flink's behaviour)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink fixYarnSlotAmending
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5532.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5532
----
commit a8bdf32b24d3f41bb8056382743b7868f597b150
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-20T10:20:41Z
[FLINK-8711] [yarn] Remove code which auto-magically changes slots per TM
The FlinkYarnSessionCli auto-magically amends the number of slots with
which the
TMs are started. This was thought as a convenience function. However, it
breaks the
execution of jobs where we have multiple slot sharing groups. Therefore,
this commit
removes this code.
As a consequence, the user has to make sure that he specifies enough slots
and
containers for the pre Flip-6 code such that the number of total slots is
greater
or equal than the parallelism of the job.
----
> Flink with YARN uses wrong SlotsPerTaskManager
> ----------------------------------------------
>
> Key: FLINK-8711
> URL: https://issues.apache.org/jira/browse/FLINK-8711
> Project: Flink
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.3.2, 1.5.0
> Reporter: Aleksandr
> Assignee: Till Rohrmann
> Priority: Critical
>
> I see wrong behavior for Flink in YARN.
> I tried to setup SlotsPerTaskManager using "-ys 2 ", but it used only 1 slot.
> I found the code
> [https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L387]
> For example, when I have :"-yn 7 -ys 2 -p 2" in log I see:
> "The YARN cluster has 14 slots available, but the user requested a
> parallelism of 2 on YARN. Each of the 7 TaskManagers will get 1 slots."
> Why can't we use -ys with -p?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
