MapSate within Aggregate function

Hi, We have SlidingProcessingTimeWindows running with AggregateFunction and Window Function. How we use MapState within AggregateFunction to keep storing incoming elements as we receive Millions of elements over 24 running sliding windows ? If we don't do that then AggregateFunction state grows b

Re: Memory constrains running Flink on Kubernetes

Thanks for all the answers so far. Especially clarifying was that RocksDB memory usage isn't accounted for in the flink memory metrics. It's clear that we need to experiment to understand it's memory usage and knowing that we should be looking at the container memory usage minus all the jvm manage

Re: LEFT JOIN issue SQL API

Thanks so much for your answer , but then how should I perform such as comparison ? Which options do we have ? Thanks Le mer. 24 juill. 2019 10:01 p.m., Ruidong Li a écrit : > Hi, it's because the Outer Joins will generate retractions, consider the > behavior of Left Outer Join > > 1. left reco

Re: Memory constrains running Flink on Kubernetes

Hi It's definitely not easy to calculate the accurate memory usage of RocksDB, but formula of "block-cache-memory + column-family-number * write-buffer-memory * write-buffer-number + index&filter memory" should give enough sophisticated hints. When talking about the column-family-number, they

Help with the correct Event Pattern

Hello everyone, I need a bit of help concerning a correct formulation for a Complex Event Pattern, using CEP. I have a stream of events which once keyed for ids, they may look like this: a b1 b2 b3 b4 b5 c1 c2 d1 d2 c3 c4 e1 e2 f1 what I want to achieve is to get, from a formulation similar to

Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？

Yes，that’s the point , FilesCreated = CreateFileOps + FsDirMkdirOp All I can say is  --thanksDarling Andrew D.Lin下面是被转发的邮件：发件人: Yun Tang 主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？日期: 2019年7月23日 GMT+8 下午4:05:42收件人: 陈Darling

Early firing window implementation issue

Hi, I am having quite hard time to understand flink windowing principals and would be very pleased if you could point me in the right direction. My purpose is to count the number of recurring events for a time interval and generate alert events if the number of recurring events is greater than a

question for handling db data

Hello, I have a question on using flink, we have a small data set which does not change often but have another data set which we need to compare with it and it has lots of data let say I have two collections geofence and locations in mongodb. Geofence collection does not change often and relative

Re: question for handling db data

Hi Jaya, Broadcast pattern may help here. Take a look at this: https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/broadcast_state.html You'll still keep your geofence data as a stream (depending on the data and use case, maybe the whole list of geofence as a single strea

Assign a Row.of(ListsElements) exception

Hello everyone, I have a list with bunch of elements and I need create a Row.of() based on the whole elements. I try to apply a lambda function for this purpose as: mylist.forEach(n->out.collect(Row.of(n))); but I got the exception below: org.apache.flink.streaming.runtime.tasks.ExceptionInChai

Re: MapSate within Aggregate function

Hi Ahmad AFAIK, Flink currently does not support stores incoming elements to a MapState, maybe the window function[1] can be help [1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/windows.html#window-functions Best, Congxian Ahmad Hassan 于2019年7月25日周四 下午5:58写

Event time window eviction

Hi All, I'm working on a very short tumbling window for 1 second per key. What I want to achieve is if the event time per key doesn't progress after a second I want to evict the window, basically a combination of event time and processing time. I'm currently achieving it by registering a processin