[
https://issues.apache.org/jira/browse/FLINK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204554#comment-16204554
]
Xingcan Cui commented on FLINK-7799:
------------------------------------
Hi [~fhueske], I've reconsidered this problem and found a drawback of the time
block solution.
Specifically, if we merge all the rows belonging to the same time block in an
entry (the value of which is either a {{Map}} or a {{List}}) of the
{{MapState}}, the minimum operating unit of the state becomes a collection.
That means everytime we store/remove a single row, all the data in the same
block must also be rewritten, which will definitely bring a lot of extra cost.
If that drawback cannot be eliminated, I wonder if we could improve the join
performance from another point of view. Since the rocksdb backend should be
widely used in real applications and the {{MapState}} entries are ordered in
it, can we provide something like a hint mechanism in the state API, so that
the join function can be aware of the ordering?
> Improve performance of windowed joins
> -------------------------------------
>
> Key: FLINK-7799
> URL: https://issues.apache.org/jira/browse/FLINK-7799
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
> Affects Versions: 1.4.0
> Reporter: Fabian Hueske
> Assignee: Xingcan Cui
> Priority: Critical
>
> The performance of windowed joins can be improved by changing the state
> access patterns.
> Right now, rows are inserted into a MapState with their timestamp as key.
> Since we use a time resolution of 1ms, this means that the full key space of
> the state must be iterated and many map entries must be accessed when joining
> or evicting rows.
> A better strategy would be to block the time into larger intervals and
> register the rows in their respective interval. Another benefit would be that
> we can directly access the state entries because we know exactly which
> timestamps to look up. Hence, we can limit the state access to the relevant
> section during joining and state eviction.
> The good size for intervals needs to be identified and might depend on the
> size of the window.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
