[
https://issues.apache.org/jira/browse/FLINK-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154300#comment-15154300
]
ASF GitHub Bot commented on FLINK-2237:
---------------------------------------
Github user ggevay commented on the pull request:
https://github.com/apache/flink/pull/1517#issuecomment-186244959
@fhueske:
So far, I have only run benchmarks on my work laptop and home laptop.
I have used the newly created `ReducePerformance`, with varying the
`(number of elements / number of different keys)` ratio, and also the memory
usage (between 250 MB - 6 GB). The hash-based strategy was always faster, up to
3 times, and mostly by a factor of 1.5 - 2.5.
I have also used `ReduceHashTableTest.testWithIntPair`. To make this a
benchmark test, the lines that deal with the `reference` hash table for
checking the correct output should be commented out, and the time measurement
commented in. (This only tests the hash table, not end-to-end; and it can't be
directly compared with the sort based reduce.)
I have also used `HashTablePerformanceComparison` to compare with the other
hash tables. This has the disadvantage that the workload here is not
characteristic of a reduce (eg. no processRecordWithReduce calls, because the
other hash tables don't have that).
I have also benchmarked using my game theory stuff (which originally
prompted me to start working on this), and the hash-based combiner was faster
here as well.
> Did you check the combine rate (input / output ratio) compared to the
sort-based strategy?
I didn't measure it directly, but `ReducePerformance` shows the difference
nicely when the number of input elements is large enough that the sort-based
strategy has to emit often, but the number of different keys is small enough
that the hash-based strategy can fit all the keys in memory. In this case, the
speedup of the final reduce phase is striking.
> Have you checked heap memory consumption / GC activity compared to the
sort-based strategy?
I haven't actually checked it, but it should be the same or better as with
the sort-based strategy:
- enabled object reuse: no allocations at all
- disabled object reuse: one allocation per input element, and one
allocation per reduce step
The sort-based strategy might additionally have allocations inside maybe
the sorter and/or MergeIterator.
> Add hash-based Aggregation
> --------------------------
>
> Key: FLINK-2237
> URL: https://issues.apache.org/jira/browse/FLINK-2237
> Project: Flink
> Issue Type: New Feature
> Reporter: Rafiullah Momand
> Assignee: Gabor Gevay
> Priority: Minor
>
> Aggregation functions at the moment are implemented in a sort-based way.
> How can we implement hash based Aggregation for Flink?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
