[
https://issues.apache.org/jira/browse/FLINK-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142753#comment-15142753
]
ASF GitHub Bot commented on FLINK-2237:
---------------------------------------
Github user ggevay commented on the pull request:
https://github.com/apache/flink/pull/1517#issuecomment-182872207
I'm thinking about partitioning, but again, I'm finding it too difficult to
imagine how this algorithm behaves, when we emit/rebuild only some part of the
table at a time. In the partitioned case, there seem to be a lot of potential
for messing things up in complicated and hard to foresee ways.
For example, if I always emit the partition with the least number of
segments, then what will actually happen is that I will kinda always emit the
same partition (because after an emit, it starts from a "disadvantage" compared
to the others), and the others just keep growing until they eat all the memory,
so this degenerates into the situation where there isn't a single memory
segment left for the smallest one and _bad things_ happen.
In contrast, an advantage of the current design is that its performance
characteristics are not that hard to grok, and it is relatively straightforward
to see how the sizes of the 3 memory regions (bucket segments, record segments,
staging segments) relate to each other.
Btw. I think I figured out a simple way to do even the final reduce with
the spilling without partitioning the memory: when memory is full I write
everything to disk, and I partition only at this time, by writing to different
files (the same files across multiple spillings, so eg. the second spilling
appends to the same files as the first one created), and then when the input
ends, I recurse on processing the files one by one (and I start a new level of
partitioning).
> Add hash-based Aggregation
> --------------------------
>
> Key: FLINK-2237
> URL: https://issues.apache.org/jira/browse/FLINK-2237
> Project: Flink
> Issue Type: New Feature
> Reporter: Rafiullah Momand
> Assignee: Gabor Gevay
> Priority: Minor
>
> Aggregation functions at the moment are implemented in a sort-based way.
> How can we implement hash based Aggregation for Flink?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
