[
https://issues.apache.org/jira/browse/FLINK-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063769#comment-15063769
]
ASF GitHub Bot commented on FLINK-7:
------------------------------------
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1255#issuecomment-165732606
Range partitioning serves two purposes:
1. producing fully sorted results.
2. evenly balancing the load in case of skewed key distributions.
Producing sorted results is working fine. However, producing balanced
partitions does not seem to work so well. Looking at the numbers I posted, the
partitions produced by the range partitioner are less balanced than the hash
partitioned ones (records-in / bytes-in). The difference is not huge, but still
range partitioning should be able to do better than hash partitioning.
I proposed to increase the sample size, because this should improve the
accuracy of the histogram without having a (measurable) impact on the
performance. If we pay so much time to generate a histogram, the histogram
should be accurate enough to result in balanced partitions.
Can you explain how you calculated the sample size of `parallelism * 20`?
> [GitHub] Enable Range Partitioner
> ---------------------------------
>
> Key: FLINK-7
> URL: https://issues.apache.org/jira/browse/FLINK-7
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Runtime
> Reporter: GitHub Import
> Assignee: Chengxiang Li
> Fix For: pre-apache
>
>
> The range partitioner is currently disabled. We need to implement the
> following aspects:
> 1) Distribution information, if available, must be propagated back together
> with the ordering property.
> 2) A generic bucket lookup structure (currently specific to PactRecord).
> Tests to re-enable after fixing this issue:
> - TeraSortITCase
> - GlobalSortingITCase
> - GlobalSortingMixedOrderITCase
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/7
> Created by: [StephanEwen|https://github.com/StephanEwen]
> Labels: core, enhancement, optimizer,
> Milestone: Release 0.4
> Assignee: [fhueske|https://github.com/fhueske]
> Created at: Fri Apr 26 13:48:24 CEST 2013
> State: open
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
