Xiaogang Shi created FLINK-5544:
-----------------------------------
Summary: Implement Internal Timer Service in RocksDB
Key: FLINK-5544
URL: https://issues.apache.org/jira/browse/FLINK-5544
Project: Flink
Issue Type: Bug
Components: Streaming
Reporter: Xiaogang Shi
Now the only implementation of internal timer service is
HeapInternalTimerService which stores all timers in memory. In the cases where
the number of keys is very large, the timer service will cost too much memory.
A implementation which stores timers in RocksDB seems good to deal with these
cases.
It might be a little challenging to implement a RocksDB timer service because
the timers are accessed in different ways. When timers are triggered, we need
to access timers in the order of timestamp. But when performing checkpoints, we
must have a method to obtain all timers of a given key group.
A good implementation, as suggested by [~StephanEwen], follows the idea of
merge sorting. We can store timers in RocksDB with the format
{{KEY_GROUP#TIMER#KEY}}. In this way, the timers under a key group are put
together and are sorted.
Then we can deploy an in-memory heap which keeps the first timer of each key
group to get the next timer to trigger. When a key group's first timer is
updated, we can efficiently update the heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
