Re: How to print the aggregated state everytime it is updated?

2020-03-10 Thread Arvid Heise
Hi Kant, if you only want to output every second, you probably want to use a ProcessFunction with timers [1]. Basically, this function holds the states and manages the updates to it. The updates should also be stored in a local/non-state variable *changes*. Whenever the timer triggers, you would

Re: How to print the aggregated state everytime it is updated?

2020-03-06 Thread Robert Metzger
Hey, I don't think you need to use a window operator for this use case. A reduce (or fold) operation should be enough: https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/ On Fri, Mar 6, 2020 at 11:50 AM kant kodali wrote: > Hi, > > Thanks for this. so how can I

Re: How to print the aggregated state everytime it is updated?

2020-03-06 Thread kant kodali
Hi, Thanks for this. so how can I emulate an infinite window while outputting every second? simply put, I want to store the state forever (say years) and since rocksdb is my state backend I am assuming I can state the state until I run out of disk. However I want to see all the updates to the

Re: How to print the aggregated state everytime it is updated?

2020-03-06 Thread Congxian Qiu
Hi >From the description, you use window operator, and set to event time. then you should call `DataStream.assignTimestampsAndWatermarks` to set the timestamp and watermark. Window is triggered when the watermark exceed the window end time Best, Congxian kant kodali 于2020年3月4日周三 上午5:11写道: >

How to print the aggregated state everytime it is updated?

2020-03-03 Thread kant kodali
Hi All, I have a custom aggregated state that is represent by Set and I have a stream of values coming in from Kafka where I inspect, compute the custom aggregation and store it in Set. Now, I am trying to figureout how do I print the updated value everytime this state is updated? Imagine I have