Re: Tag flink metrics to job name

Is there a way I can look into say for a specific job what’s the cpu usage or memory usage of the yarn containers when multiple jobs are running on the same cluster. Also, the issue am trying to resolve is I’m seeing high memory usage for one of the containers I want isolate the issue with one job

Flink 1.11.3 not able to restore with savepoint taken on Flink 1.9.3

Hi, We are trying to upgrade Flink from version 1.9.3 to 1.11.3. As part of the upgrade testing, we are observing below exception when Flink 1.11.3 tries to restore from a savepoint taken with Flink 1.9.3. java.lang.Exception: Exception while creating StreamOperatorStateContext. at org.ap

Re: Tag flink metrics to job name

hmm...in a roundabout way this could be possible I suppose. For a given job, search through your metrics for some job metric (like numRestarts on the JM, or any task metric for TMs), and from that you should be able to infer the JM/TM that belongs to that (based on the TM ID / host information

RE: latency related to the checkpointing mode EXACTLY ONCE

Many thanks for your quick response. The config read_commit for the kafka consumers is required by the exactly once (EOS)? No exactly once if we read un committed messages? Regards, Min From: Chesnay Schepler Sent: Thursday, February 18, 2021 8:27 PM To: Tan, Min ; user Subject: [External] Re

Re: Flink 1.11.3 not able to restore with savepoint taken on Flink 1.9.3

Hi, I'm not aware of any breaking changes in the savepoint formats from 1.9.3 to 1.11.3. Let's first try to rule out any obvious causes of this: - Were any data types / classes that were used in state changed across the restores? Remember that keys types are also written as part of state snapshot

Re: How is proctime represented?

hmm...I can now see where that uncertainty comes from. My /impression/ is that PROCTIME is not evaluated eagerly, and instead and operators relying on this column generate their own processing timestamp. What throws me off is that I cannot tell how you would tell Flink to store a processing ti

AW: Kafka SQL Connector: dropping events if more partitions then source tasks

If i increase the watermark, the dropped events getting lower. But why is the DataStream API Job still running with 12 hours watermark delay? By the way, I'm using Flink 1.11. It would be nice if someone could give me some advice. Best, Jan Von: Jan Oelschlegel Gesendet: Donnerstag, 18. Febru

Re: How is proctime represented?

Chesnay is right. The PROCTIME() is lazy evaluated and executed when its result is needed as an argument for another expression or function. So within the pipeline the column is NULL but when you want to compute something e.g. CAST(proctime AS TIMESTAMP(3)) it will be materialized into the row.

Flink problem

hi all      For example, if A user message A (uerId: 001) is reported, and no record is reported again by userId: 001 within 10 minutes, record A will be sent out. How can this be achieved in Flink? Thanks Jiazhi 发自我的iPhone

Re: State Access Beyond RichCoFlatMapFunction

Hello, Is there an example setup of Queryable State for a Local Embedded Environment? I am trying to execute Flink programs from within IntelliJ. Any help would be appreciated! Even if not, if there are other examples where QueryableState can be executed in a standalone cluster, that would als

Re: How is proctime represented?

Thanks yall this is really helpful! On Fri, Feb 19, 2021 at 2:40 AM Timo Walther wrote: > Chesnay is right. The PROCTIME() is lazy evaluated and executed when its > result is needed as an argument for another expression or function. So > within the pipeline the column is NULL but when you want t

Re: How do I increase number of db connections of the Flink JDBC Connector?

Ah got it, thanks! On Thu, Feb 18, 2021 at 10:53 PM Chesnay Schepler wrote: > Every works uses exactly 1 connection, so in order to increase the > number of connections you must indeed increase the worker parallelism. > > On 2/19/2021 6:51 AM, Li Peng wrote: > > Hey folks, > > > > I'm trying to

How to pass PROCTIME through an aggregate

Hello, Using the table api, I have a CREATE DDL which adds a PROCTIME() column and I need to use it deep (like 10 operators deep) in our plan. However, I explicitly do not want to use it before that point. The problem I'm running into is for any group by UDAF I use I then lose the proctime column

Re: Flink problem

What you're looking for might be Session Window[1]. Thank you~ Xintong Song [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/operators/windows.html#session-windows On Fri, Feb 19, 2021 at 7:35 PM ゞ野蠻遊戲χ wrote: > hi all > > For example, if A user message A (uer

Re: Flink problem

Can you tell what to do when the record is reported again by userId:001 within 10 minutes, for example buffer it or keep the only one ？ ゞ野蠻遊戲χ 于2021年2月19日周五 下午7:35写道： > hi all > > For example, if A user message A (uerId: 001) is reported, and no > record is reported again by userId: 001 wit

Re: Question

Hi, I read it: https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/local_installation.html I can run the code in the UI of Apache Flink that is in the bin file of Apache Flink. If I run a java code from intellij idea or eclipse, then how can I connect the code to apache flink U

Cep application with Flink

Hi there.   I’m a newby with Flink and I need to develop a CEP application using this technology. At least, we are choosing a tool now. So I have a question how Flink really fit to my goals.   Buisness requirements:   There is a mobile application. Data is collected in Kafka topics (which are h