Flink memory usage

Hi, I would like to share some considerations about Flink memory consumption. I have a cluster composed of three nodes: 1 used both as JM and TM and other 2 TM. I ran two identical applications (in different moments) on it. The only difference is that on the second one I doubled every operators,

Re: FlinkCEP, circular references and checkpointing failures

Thank you very much for your steady response, Kostas! Cheers, Federico 2017-11-03 16:26 GMT+01:00 Kostas Kloudas : > Hi Federico, > > Thanks for trying it out! > Great to hear that your problem was fixed! > > The feature freeze for the release is going to be next

Re: Making external calls from a FlinkKafkaPartitioner

Thanks Nico - Thanks for the feedback, and nice catch on the missing volatile. Ron — Ron Crocker Principal Engineer & Architect ( ( •)) New Relic rcroc...@newrelic.com M: +1 630 363 8835 > On Nov 3, 2017, at 7:48 AM, Nico Kruber wrote: > > Hi Ron, > imho your code

Re: Using Flink Ml with DataStream

Hi Chesnay, Thanks for the reply, do you know how to serve using the trained model? Where is the model saved? Regards, Adarsh ‌ On Wed, Nov 1, 2017 at 4:46 PM, Chesnay Schepler wrote: > I don't believe this to be possible. The ML library works exclusively with > the

Re: Incremental checkpointing documentation

Hi Elias, let me answer the questions to the best of my knowledge, but in general I think this is as expected. (Let me give a link to the docs explaining the activation [1] for other readers first.) On Friday, 3 November 2017 01:11:52 CET Elias Levy wrote: > What is the interaction of

Re: Making external calls from a FlinkKafkaPartitioner

Hi Ron, imho your code should be fine (except for a potential visibility problem on the changes of the non-volatile partitionMap member, depending on your needs). The #open() method should be called (once) for each sink initialization (according to the javadoc) and then you should be fine with

Re: FlinkCEP, circular references and checkpointing failures

Perfect! thanks a lot! Kostas > On Nov 3, 2017, at 3:23 PM, Federico D'Ambrosio > wrote: > > Hi Kostas, > > yes, I'm using 1.3.2. I'll try the current master and I'll get back to you. > > 2017-11-03 15:21 GMT+01:00 Kostas Kloudas

Re: FlinkCEP, circular references and checkpointing failures

Hi Kostas, yes, I'm using 1.3.2. I'll try the current master and I'll get back to you. 2017-11-03 15:21 GMT+01:00 Kostas Kloudas : > Hi Federico, > > I assume that you are using Flink 1.3, right? > > In this case, in 1.4 we have fixed a bug that seems similar to

Re: FlinkCEP, circular references and checkpointing failures

Hi Federico, I assume that you are using Flink 1.3, right? In this case, in 1.4 we have fixed a bug that seems similar to your case: https://issues.apache.org/jira/browse/FLINK-7756 Could you try the current master to see if it fixes your

Re: FlinkCEP, circular references and checkpointing failures

I'm sorry, I realized that the stacktrack was poorly formatted, here it is a better formatting: AsynchronousException{java.lang.Exception: Could not materialize checkpoint 2 for operator KeyedCEPPatternOperator -> alert-select -> Sink: notification-sink-1 (1/1).} at

FlinkCEP, circular references and checkpointing failures

Hello everyone, I'm a bit experimenting with FlinkCEP and I'm noticing weird failures when it comes to checkpoints and within clauses windows closing at the same time a (synchronous, both on Fs and RocksDB, stored in hdfs) checkpoint occurs. The following is the relevant code: val env :

Re: Negative values using latency marker

Hi Tovi, if I see this correctly, the LatencyMarker gets its initial timstamp during creation at the source and the latency is reported as a metric at a sink by comparing the initial timestamp with the current time. If the clocks between the two machines involved diverge, e.g. the sinks clock

Fwd: Initialise side input state

Hi Xingcan, On Fri, Nov 3, 2017 at 3:38 AM, Xingcan Cui wrote: > Hi Maxim, > > if I understand correctly, you actually need to JOIN the fast stream with > the slow stream. Could you please share more details about your problem? > Sure I can explain more, with some example