GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/1147
Windows
Adds new dedicated implementations for grouped windows on aligned
processing time (tumbling and sliding time windows). A lot of code is written
to be easily adaptable to event-time implementations.
The pull request currently includes only the operator implementations - the
operators are not yet integrated with the APIs and stream graph builder.
The new operators fix many problems that the current time operators have,
including cleanup of stale keys, deadlocks, resource leaks, incorrect time
alignment, and occasional wrong results.
The pull request also adds quite a few tests for the operators and data
structures, including checks whether data structures are released and whether
threads are properly disposed.
One can try out the implementation by running the
`GroupedProcessingTimeWindowExample` that I added for experimental purposes
into the examples project (should be removed before merging).
#### Speed
The new implementations are also much faster:
Setup is a streaming program that generates (Long, Long) tuples, and groups
them (field 0) in a window of 2500 msecs sliding by 500 msecs, and aggregates
them (field 1). I use 10k unique keys in the example
Times:
- current implementation: 50K / core / sec (gets slower over time, high GC
overhead)
- new implementation w/o pre-aggregation: 800K / sec / core (moderate GC
overhead)
- new implementation w/ pre-aggregation: 3mio / sec / core (low GC
overhead)
This build on #1133
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink windows
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1147
----
commit 3c113064826612d64bc6902733511a5f332543cd
Author: Stephan Ewen <[email protected]>
Date: 2015-09-15T12:55:41Z
[FLINK-2675] [streaming] Add utilities for scheduled triggers.
commit 1654189e772e55c04175374ec878ba398e2c9fbf
Author: Stephan Ewen <[email protected]>
Date: 2015-09-15T13:00:17Z
[FLINK-2683] [FLINK-2682] [runtime] Add dedicated operator for aligned
processing time windows.
Also add utilities for heap-backed keyed state in panes (dedicated tailored
hash table)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
