[ https://issues.apache.org/jira/browse/FLINK-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965367#comment-14965367 ]
ASF GitHub Bot commented on FLINK-2834: --------------------------------------- GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/1272 [FLINK-2834] Global round-robin for temporary directories Multiple TaskManager filesystems can be used by configuring multiple temporary directories. This patch changes the process of spilling files from a per-operator round-robin to a global round-robin such that each directory is written to in turn across all operators, reducing unbalanced I/O due to bunching. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 2834_global_round_robin_for_temporary_directories Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1272.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1272 ---- commit 4af4332bfdf63112e940f706a6b8fedc0b7cff5b Author: Greg Hogan <c...@greghogan.com> Date: 2015-10-20T14:47:58Z [FLINK-2834] Global round-robin for temporary directories Multiple TaskManager filesystems can be used by configuring multiple temporary directories. This patch changes the process of spilling files from a per-operator round-robin to a global round-robin such that each directory is written to in turn across all operators, reducing unbalanced I/O due to bunching. ---- > Global round-robin for temporary directories > -------------------------------------------- > > Key: FLINK-2834 > URL: https://issues.apache.org/jira/browse/FLINK-2834 > Project: Flink > Issue Type: Improvement > Components: Distributed Runtime > Affects Versions: 0.10 > Reporter: Greg Hogan > Assignee: Greg Hogan > Priority: Minor > > Currently each {{FileIOChannel}} performs a local round-robin among temporary > directories. Initially every writer will be concurrently spilling to the same > first directory, then the same second directory, etc. Even when evenly > dispersed this effect will naturally occur due to the bunching effect. > One solution is to use a global counter when selecting the temporary path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)