[
https://issues.apache.org/jira/browse/FLINK-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141158#comment-15141158
]
ASF GitHub Bot commented on FLINK-2237:
---------------------------------------
Github user ggevay commented on the pull request:
https://github.com/apache/flink/pull/1517#issuecomment-182477373
> In any case, we need to update the documentation for the added
CombineHint. I would also be good to extend ReduceITCase with a few tests that
use CombineHint.HASH.
OK, I'll do these.
> the full stop to resize / rebuild / emit records might take some time,
depending on the size of the table.
Emit is very fast, because `EntryIterator` is reading the record area
sequentially. I don't think there is much room for improvement there. (Except
perhaps your idea of storing the lengths as negative values, and skipping over
abandoned records :))
__Rebuilds because of compactions:__
Typically, the size of the records don't change, in which case no
compactions happen.
If the records change size, that's a mess. But even in this case,
performance should be reduced by less than 2x, because the number of insertions
done in rebuilds should be proportional to the number of normal update/insert
operations, and one insert during rebuild is faster then one usual
insert/update, because the linked lists are not traversed (only the bucket
segments are accessed randomly; the elements are inserted to the beginning of
the lists).
__Rebuilds because of resizing__:
If the serializer knows the length of the records in advance, then no
resizings will happen, because we just calculate the final number of bucket
segments at the beginning (see `calcInitialNumBucketSegments`). Unfortunately,
currently the serializers almost never know the length, but I hope this
situation will improve in the future (for example, see FLINK-3321).
If the serializer doesn't know the length, then how much time is spent with
resizings is greatly affected by the average number of records per key. This is
because the time spent with doing resizes is proportional to the number of
elements inserted, and if there are lots of elements with matching keys, then
most operations will be updates (rather than insertions). I will probably do
some performance tests to confirm this, and see how much time is spent in
resizes in the worst case (when every key has only one element). (Also, the
"performance should be reduced by less than 2x" argument from above should be
applicable in this case as well.)
> Do you think it is possible to extract the ReduceFunction from the table?
This should be doable, I'll try.
> Should we think about linear hashing for resizing the table.
Thanks, I didn't know about this algorithm. I'll think about this.
> Split the single record into multiple partitions (i.e., use multiple
RecordAreas). Each partition holds the data of multiple buckets. This allows to
restrict rebuilding, compaction, etc. to only a part of the whole table.
Do you think that this would have a performance benefit? The time of a
rebuild is linear with the size, so I don't see how doing it in chunks would
improve performance. (I mean, the (frequency of rebuilds of a partition) /
(frequency of rebuilds of the full table) is probably the same ratio as the
(time to rebuild a partition) / (time to rebuild the full table).)
> It can also help to make your implementation more suitable for the final
reduce case
Yes, some partitioning will be needed for that. However, I would like to
get this in first, and then I'll design the implementation of that case.
> Add hash-based Aggregation
> --------------------------
>
> Key: FLINK-2237
> URL: https://issues.apache.org/jira/browse/FLINK-2237
> Project: Flink
> Issue Type: New Feature
> Reporter: Rafiullah Momand
> Assignee: Gabor Gevay
> Priority: Minor
>
> Aggregation functions at the moment are implemented in a sort-based way.
> How can we implement hash based Aggregation for Flink?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
