[
https://issues.apache.org/jira/browse/FLINK-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965367#comment-14965367
]
ASF GitHub Bot commented on FLINK-2834:
---------------------------------------
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/1272
[FLINK-2834] Global round-robin for temporary directories
Multiple TaskManager filesystems can be used by configuring multiple
temporary directories.
This patch changes the process of spilling files from a per-operator
round-robin to a global
round-robin such that each directory is written to in turn across all
operators, reducing
unbalanced I/O due to bunching.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
2834_global_round_robin_for_temporary_directories
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1272.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1272
----
commit 4af4332bfdf63112e940f706a6b8fedc0b7cff5b
Author: Greg Hogan <c...@greghogan.com>
Date: 2015-10-20T14:47:58Z
[FLINK-2834] Global round-robin for temporary directories
Multiple TaskManager filesystems can be used by configuring multiple
temporary directories.
This patch changes the process of spilling files from a per-operator
round-robin to a global
round-robin such that each directory is written to in turn across all
operators, reducing
unbalanced I/O due to bunching.
----
> Global round-robin for temporary directories
> --------------------------------------------
>
> Key: FLINK-2834
> URL: https://issues.apache.org/jira/browse/FLINK-2834
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
> Affects Versions: 0.10
> Reporter: Greg Hogan
> Assignee: Greg Hogan
> Priority: Minor
>
> Currently each {{FileIOChannel}} performs a local round-robin among temporary
> directories. Initially every writer will be concurrently spilling to the same
> first directory, then the same second directory, etc. Even when evenly
> dispersed this effect will naturally occur due to the bunching effect.
> One solution is to use a global counter when selecting the temporary path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
