[
https://issues.apache.org/jira/browse/FLINK-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214144#comment-14214144
]
ASF GitHub Bot commented on FLINK-1237:
---------------------------------------
GitHub user StephanEwen opened a pull request:
https://github.com/apache/incubator-flink/pull/207
[FLINK-1237] Add support for custom partitioners
- Functions: GroupReduce, Reduce, Aggregate on UnsortedGrouping,
SortedGrouping (Java API & Scala API)
- Manual partition on DataSet (Java API & Scala API)
- Distinct operations provide semantic properties for preservation of
distinctified fields
- Tests for pushown (or not pushdown) of custom partitionings and forced
rebalancing
- Tests for GlobalProperties matching of partitionings
- Caching of generated requested data properties for unary operators
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink partitioner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/207.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #207
----
commit 8a955e51959fbf3a3028496f809f89b60c4e7945
Author: Stephan Ewen <[email protected]>
Date: 2014-11-17T00:04:12Z
[FLINK-1233] Fix flaky AggregateITCase
commit 8d97af04ebf3bd530d8d0382744b5261615f18b3
Author: Stephan Ewen <[email protected]>
Date: 2014-11-13T15:26:07Z
[FLINK-1237] Add support for custom partitioners
- Functions: GroupReduce, Reduce, Aggregate on UnsortedGrouping,
SortedGrouping (Java API & Scala API)
- Manual partition on DataSet (Java API & Scala API)
- Distinct operations provide semantic properties for preservation of
distinctified fields
- Tests for pushown (or not pushdown) of custom partitionings and forced
rebalancing
- Tests for GlobalProperties matching of partitionings
- Caching of generated requested data properties for unary operators
----
> Add custom partitioners for shuffle steps
> -----------------------------------------
>
> Key: FLINK-1237
> URL: https://issues.apache.org/jira/browse/FLINK-1237
> Project: Flink
> Issue Type: Improvement
> Reporter: Till Rohrmann
> Assignee: Stephan Ewen
>
> It would be helpful to be able to specify a custom partitioner for shuffle
> steps. This would users allow to control how their data is distributed across
> the cluster. Especially in situations where the number of data records is
> equal to the number of slots it might be a possibility to avoid skewed data
> distributions due to the default hashing function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
