flink_taskmanager_job_task_operator_records_lag_max == -Inf on Flink 1.4.2

Hey guys, I just updated our Flink install from 1.4.0 to 1.4.2, but our Prometheus monitoring is not getting the current Kafka lag. After updating to 1.4.2 and making the symlink between opt/flink-metrics-prometheus-1.4.2.jar to lib/, I got the metrics back on Prometheus, but the most important

A question about Kryo and Window State

I have a running pipe with Window State in a class say Class A{ long a; } It uses the default KryoSerializer I want to add a field to Class A { long a; long b; } I need to suspend with SP and resume with the new version of Class A Is there a definite way to do this. I tried Class

Ask about writing join streams results to CSV file

Hi all, Could you help me, I joined two streams and then I tried to write the results to CSV file. But I found the CSV file is empty even the join function is called when I used use print instead of write ** here empty file ProximityStream.join(gpsStream)

Re: Clarity on Flink 1.5 Rescale mechanism

Hi, it means that you can now modify the parallelism of a running job with a new „modify“ command in the CLI, see [1]. Adding task manager will add their offered slots to the pool of available slots, it will not automatically change the parallelism. Best, Stefan [1]

Re: DataStreamCalcRule grows beyond 64 KB

Thanks Hequn for pointing out a work around for that, will try that out for now. And Timo, I am using 1.3.3. We are in the middle of upgrading Flink, so will try on 1.5 once that happens and report back. Thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Clarity on Flink 1.5 Rescale mechanism

Hello In flink 1.5 release notes - https://flink.apache.org/news/2018/05/25/release-1.5.0.html#release-notes Various Other Features and Improvements: Applications can be rescaled without manually triggering a savepoint. Under the hood, Flink will still take a savepoint, stop the application, and

Re: DataStreamCalcRule grows beyond 64 KB

Hi, which version of Flink are you using? This issue is not entirely fixed but the most important cases have been solved in Flink 1.5. See https://issues.apache.org/jira/browse/FLINK-8274. Regards, Timo Am 12.06.18 um 03:52 schrieb Hequn Cheng: Hi rakeshchalasani, At the moment flink

Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

Hi, > Can you elaborate that a little bit? are you referring to > "Output> output" in AbstractStreamOperator class? Yes. However I have never tried it, so I’m not 100% sure there are no pit falls with that. Regarding processing time timers. You should be able to register the timer once and

Re: Conceptual question

Tony, You are right; I hadn't thought this through. The KeyedStateFunction only has access to one piece of state at a time, which is the state for some specific key associated with one specific state descriptor. You can fetch, update, or clear that piece of state, but as far as I know, you can't

Re: Implementation of ElasticsearchSinkFunction, how to handle class level variables

Hi Jayant, Yes, you don’t have to use an anonymous class for the sink function. An actual separate class works just as fine. The class fields don’t need to be marked as transient or checkpointed, since they should just be constants that come with instantiation of the sink function, or could

Re: Conceptual question

Hi David, I have read the document for `Context.applyToKeyedState()`, but I still have some questions for using it to implement keyed state migration. `Context.applyToKeyedState()` can only be called in `processBoradcaseElement()`, so it won't have any key information. It looks like I can use

Re: How to submit two Flink jobs to the same Flink cluster?

Hi Angelica, The Flink cluster needs to provide a sufficient number of slots to process the tasks of all submitted jobs. Besides that there is no limit. However, if you run super many jobs, you might need to tune a few configuration parameters. Best, Fabian 2018-06-12 8:46 GMT+02:00 Sampath

Re: Hundreds of parallel jobs on Flink Cluster

Hi Chirag, if you give your Flink cluster enough resources, especially slots, to execute all of the 100 parallel jobs, then it should not be a big problem. A Flink session cluster can run multiple jobs in parallel as long as you it has enough slots to deploy the task to. There are some tuning

Re: why BlobServer use ServerSocket instead of Netty's ServerBootstrap?

Hi Makeyang, it has historical reasons why the BlobServer is using server sockets. It is just how it was first developed and we never had the capacities and need to change it. Cheers, Till On Tue, Jun 12, 2018 at 4:32 AM makeyang wrote: > thanks > > > > -- > Sent from: >

Re: Flink 1.4.0 release commit

The commit is not in the master branch but there is a tag with it [1]. [1] https://github.com/apache/flink/tree/release-1.4.0 Cheers, Till On Tue, Jun 12, 2018 at 4:40 AM Abdul Qadeer wrote: > Hi! > > I was trying to find out the commit from which 1.4.0 was released. The > release was on 29th

Re: How to submit two Flink jobs to the same Flink cluster?

Hi Angelica You can run any number of flink jobs in flink cluster. There is no restriction as such until and unless there are issues with flink jobs resource sharing(Ex : two jobs accessing same port). On Tue, Jun 12, 2018 at 5:03 AM, Angelica wrote: > I have a Flink Standalone Cluster based