Hi Jingsong, Sorry for the delayed response; this email ended up being misclassified by my mail server and I missed it. Maybe Kenn or Aljoscha has suggestions on how runners can best implement timers?
Dan On Thu, Jan 19, 2017 at 9:55 PM, lzljs3620320 <[email protected]> wrote: > Hi there, > I'm working on the beam integration for an internal system at Alibaba. Now > most of the runners put timers in memory, such as Flink, Apex, etc. (I do not > know > the implementation of Google Dataflow).But in our scene, unbounded data > has a large number of keys，which will lead to OOM(timers in memory). So > we want to store timers in state(RocksDb in disk).The problem is how to > extract fired event time timers when refresh the input > watermark. Do we have to scan all keys and timers(Now timer is composed of > Key, id, namespace, timestamp, domain)？Is there a better > implement? I'm wondering if you could give me some advice on how to implement > timers in state efficiently. Thank you! > Best,Jingsong Lee
