Re: [DISCUSS] Make Managed Memory always off-heap (Adjustment to FLIP-49)

Sorry, I just realized that I've send my feedbacks to Jingsong's email address, instead of the dev / user mailing list. Please find my comments below. Thank you~ Xintong Song On Wed, Nov 27, 2019 at 4:32 PM Xintong Song wrote: > As a participant of the discussion yesterday, I'm +1 for the pr

Emit intermediate accumulator state of a session window

*Emit intermediate accumulator(AggregateFunction ACC value) state of a session window when new event arrives* AggregateFunction#getResults() is called only when window completes. My need is emit intermediate accumulator values(result of AggregateFunction#add()) as well and write them to Sink. Bo

Re: Flink 1.9.1 KafkaConnector missing data (1M+ records)

Thank you for your reply, Some clarification: We have configured the BucketAssigner to use the *Kafka record timestamp*. Exact bucketing behavior as follows: private static final DateTimeFormatter formatter = DateTimeFormatter .ofPattern("-MM-dd'T'HH"); @Override public String getBucketId(Ka

Side output question

Hi: I am replacing SplitOperator in my flink application with a simple processor with side outputs. My questions is that does the main stream from which we get the side outputs need to have any events (ie, produced using by the using collector.collect) ?  Or can we have all the output as side o

Re: Using MapState clear, put methods in snapshotState within KeyedCoProcessFunction, valid or not?

Hi Salva As I pointed out, it's not clear for your program logic if you call 'state.clear()' within 'snapshotState' as you do not know what exact current key is. Hence, I think your idea like that does not make any sense. From my point of view, 'clear' works sometimes in your code is not a bug

Re: Firing timers on ProcessWindowFunction

Thanks Alexander, Will do. Cheers On Mon, Dec 2, 2019 at 3:23 PM Alexander Fedulov wrote: > *This Message originated outside your organization.* > -- > Hi Avi, > > In this situation I would propose to step back and use a lower level API > - ProcessFunction. You can

Re: Using MapState clear, put methods in snapshotState within KeyedCoProcessFunction, valid or not?

Hi Yun, Thanks for your reply. You mention that " ‘snapshotState’ and ‘initializeState’ interfaces are used mainly to snapshot and initialize for operator state" but..."mainly" is not "exclusively" right? So, I guess my question tries to figure out whether doing something like this is valid/make

Re: Using MapState clear, put methods in snapshotState within KeyedCoProcessFunction, valid or not?

Thanks Congxian. From what I've read, it seems that using the keyed state in `snapshotState` is incorrect...what confuses me is that if I do something like this ``` override def snapshotState(context: FunctionSnapshotContext): Unit = { if (models.nonEmpty) { modelsBytes.clear() // This

Re: Firing timers on ProcessWindowFunction

I think the only way to do this is to add keyed operator down the stream that will hold the global state. not ideal but I don't see any other option On Mon, Dec 2, 2019 at 1:43 PM Avi Levi wrote: > Hi Vino, > I have a global state that I need to mutate every X hours (e.g clean that > state or up

Re: Firing timers on ProcessWindowFunction

Hi Vino, I have a global state that I need to mutate every X hours (e.g clean that state or update its value) . I thought that there might be an option to set a timer user the timerService with it's own time interval detached from the window interval interval . On Mon, Dec 2, 2019 at 10:59 AM vino

Re: Temporary failure in name resolution on JobManager

Thanks Yang. We did try both those properties and it didn't fix it. However, we did EVENTUALLY (after some late nights!) track the issue down, not to DNS resolution but rather an obscure bug our our connector code :-( Thanks for your response, /David/ On Mon, Dec 2, 2019 at 3:16 AM Yang Wang w

Re: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Hi, We encountered similar issues that the task manager kept being killed by yarn. - flink 1.9.1 - heap usage is low. But our job is a **streaming** job, so I want to ask if this issue is only related to **batch** job or not? Thanks! Best, Victor yingjie 于2019年11月28日周四 上午11:43写道： > Piotr is

RE: Access to CheckpointStatsCounts

Many thanks for sending your reply. It is not for monitoring but for configuration. For a job starting from an empty status, we like to load the fresh configurations. For a job recovering from a checkpoint, we like to rely on the checkpoint. Regards, Min From: vino yang [mailto:yanghua1...@gm

Re: Access to CheckpointStatsCounts

Hi min, If it is only for monitoring purposes, you can just use checkpoint REST API[1] to do this work. [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/rest_api.html#jobs-jobid-checkpoints Best, Vino 于2019年12月2日周一 下午5:01写道： > Hi, > > > > Just wonder how to access t

Access to CheckpointStatsCounts

Hi, Just wonder how to access the CheckpointStatsCoutns from the main method of a job? We need to detect if a job recovers from a checkpoint or starts from an empty status. Regards, Min E-mails can involve SUBSTANTIAL RISKS, e.g. lack of confidentiality, potential manipulation of contents a

Re: Firing timers on ProcessWindowFunction

Hi Avi, Firstly, let's clarify that the "timer" you said is the timer of the window? Or a timer you want to register to trigger some action? Best, Vino Avi Levi 于2019年12月2日周一 下午4:11写道： > Hi, > Is there a way to fire timer in a ProcessWindowFunction ? I would like to > mutate the global state

Firing timers on ProcessWindowFunction

Hi, Is there a way to fire timer in a ProcessWindowFunction ? I would like to mutate the global state on a timely basis.