GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5923
[FLINK-9253][network] make the maximum floating buffers count channel-type
independent
## What is the purpose of the change
The credit-based flow control path assigns exclusive buffers only to remote
channels (which makes sense since local channels don't use any own buffers).
However, this is a bit intransparent with respect to how much data may be in
buffers since this depends on the actual schedule of the job and not the job
graph.
By adapting the floating buffers to use a maximum of `numberOfInputChannels
* networkBuffersPerChannel + extraNetworkBuffersPerGate - #exclusiveBuffers`,
we would be channel-type agnostic and keep the old behaviour of using at most
`numberOfInputChannels * networkBuffersPerChannel + extraNetworkBuffersPerGate`
buffers in total.
## Brief change log
- change the floating buffer pool to use the remaining number of buffers
from a total maximum of `numberOfInputChannels * networkBuffersPerChannel +
extraNetworkBuffersPerGate` per channel
## Verifying this change
This change is already covered by existing tests, such as
`NetworkEnvironmentTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
(well, kind of, but not per record)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-9253
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5923.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5923
----
commit dfd85bc2cb776d00c9fac1c2844b1c4d6b19f568
Author: Nico Kruber <nico@...>
Date: 2018-04-26T16:38:00Z
[FLINK-9253][network] make the maximum floating buffers count channel-type
independent
Although with credit-based flow control exclusive buffers are only assigned
to
remote input channels, let's keep the overall maximum number of buffers per
(bounded) input gate at
numberOfInputChannels * networkBuffersPerChannel +
extraNetworkBuffersPerGate
This is easier to grasp for the user than a buffer count that depends on the
deployment.
----
---
