NPE when aggregate window.

Hi, I encounter a weird NPE when try to do aggregate on a fixed window. If I set a small parallism number the whole job uses only one TaskManager, this NPE will not happen. But when the job scales to two TaskManagers, the TaskManager will crash at Create stage. The Flink version I use is 1.11.1.

Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode

Hello Yang, It is very kind of you to give such a detailed explanation! Thanks for clarification. For the small document fix I mentioned, what do you think? Best, Fuyao From: Yang Wang Date: Monday, April 12, 2021 at 23:03 To: Fuyao Li Cc: user , Yan Wang Subject: [External] : Re: Conflict

Re: Conflict in the document - About native Kubernetes per job mode

Hi Fuyao, Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode. In your attached video, it is a PoC implementation fo

Re: how to convert DataStream to Table

Hi, Here's an example that works for me: """ import org.apache.flink.api.common.typeinfo.Types; import org.apache.flink.api.java.typeutils.RowTypeInfo; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.bridge.java.StreamTableEnvironme

Conflict in the document - About native Kubernetes per job mode

Hello Community, Yang, I noticed a conflict in the document for per-job mode support for Kubernetes. In the doc here [1], it mentions in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available

Re: Manage Kafka Offsets for Fault Tolerance Without Checkpointing

Hi Roman, Arvid, So, to achieve "at least once" guarantee, currently, automatic restart of Flink should be disabled? Is there any workaround to get "at least once" semantics with Flink Automatic restarts in this case? Regards, Rahul On Mon, Apr 12, 2021 at 9:57 PM Roman Khachatryan wrote: > Hi

Re: Reducing Task Manager Count Greatly Increases Savepoint Restore

That's really helpful, thanks Till! On Thu, Apr 8, 2021 at 6:32 AM Till Rohrmann wrote: > Hi Kevin, > > when decreasing the TaskManager count I assume that you also decrease the > parallelism of the Flink job. There are three aspects which can then cause > a slower recovery. > > 1) Each Task get

Re: How to know if task-local recovery kicked in for some nodes?

Hi Till, Got it, that definitely makes sense, was just looking for some ballpark number to start with. Appreciate your help! Thanks, Sonam From: Till Rohrmann Sent: Monday, April 12, 2021 1:00 AM To: Sonam Mandal Cc: dhanesh arole ; Tzu-Li (Gordon) Tai ; user@

Re: Manage Kafka Offsets for Fault Tolerance Without Checkpointing

Hi, Thanks for the clarification. > Other than managing offsets externally, Are there any other ways to guarantee > "at least once" processing without enabling checkpointing? That's currently not possible, at least with the default connector. Regards, Roman On Mon, Apr 12, 2021 at 3:14 PM Rah

Re: Query regarding flink metric types

Hi Suchithra, You are right, those metrics can only grow, at least until failover. isBackPressured is reported as a boolean on subtask level. These samples are then aggregated and a ratio of (times-back-pressured / number-of-samples) is reported to the JobManager. Regards, Roman On Fri, Apr 9,

Python Integration with Ververica Platform

I've been using the Community Edition v2.4. Just wondering if there is a python integration coming in future versions. tnanks -- Robert Cullen 240-475-4490

Re: Flink 1.11.4?

Hi Maciek, There are no specific plans for 1.11.4 yet as far as I know. The official policy is to support the current and previous minor release [1]. So 1.12 and 1.13 will be officially supported once 1.13 is released. However, it's likely that 1.11.4 will still be released. [1] https://flink.apa

Re: Manage Kafka Offsets for Fault Tolerance Without Checkpointing

Hi Roman, Thanks for the reply. This is what I meant by Internal restarts - Automatic restore of Flink Job from a failure. For example, pipeline restarts when Fixed delay or Failure

Re: Flink Metric isBackPressured not available

Hi, The metric is registered upon task deployment and reported periodically. Which Flink version are you using? The metric was added in 1.10. Are you checking it in the UI? Regards, Roman On Fri, Apr 9, 2021 at 8:50 PM Claude M wrote: > > Hello, > > The documentation here > https://ci.apache.

Re: Manage Kafka Offsets for Fault Tolerance Without Checkpointing

Hi, Could you please explain what you mean by internal restarts? If you commit offsets or timestamps from sink after emitting records to the external system then there should be no data loss. Otherwise (if you commit offsets earlier), you have to persist in-flight records to avoid data loss (i.e.

Manage Kafka Offsets for Fault Tolerance Without Checkpointing

Hello, *Context*: We have a stateless Flink Pipeline which reads from Kafka topics. The pipeline has a Windowing operator(Used only for introducing a delay in processing records) and AsyncI/O operators (used for Lookup/Enrichment). "At least Once" Processing semantics is needed for the pipeline

Re: how to convert DataStream to Table

Hi, I'm pulling in Timo and Jark as they know Table API better. Regards, Roman On Sun, Apr 11, 2021 at 3:36 PM vtygoss wrote: > > Hi All, > > > there is a scenario where I need to process OGG Log data in kafka using Flink > Sql. I can convert the OGG Log Stream to DataStream and each event >

Flink 1.11.4?

Hello, I'd like to ask if there are any plans to release 1.11.4 - I understand it will be last bugfix release for 1.11.x branch, as 1.13.0 is "just round the corner"? There are a few fixes we'd like to use - e.g. https://issues.apache.org/jira/browse/FLINK-9844, https://issues.apache.org/ji

Re: Re: Does it support rate-limiting in flink 1.12?

In Flink, you can only limit memory usage, e.g. via taskmanager.memory.process.size [1] (throttling could be implemented using the DataStream API, but you mentioned you are using SQL). Quotas on other resources can be set in the underlying resource manager. But I'd suggest investigating the failur

Re: How to know if task-local recovery kicked in for some nodes?

Hi Sonam, The state size probably depends a bit on your infrastructure. Assuming you have 1 GBps network connection and local SSDs, then I guess you should see a difference if your local state size is > 1 GB. Cheers, Till On Wed, Apr 7, 2021 at 1:46 PM Sonam Mandal wrote: > Hi Till and Dhanes

Re: Does it support rate-limiting in flink 1.12?

Hi, I'm not sure that I fully understand your question. Is the intention to prioritize some jobs over the others in the same Flink cluster? Currently, it is not possible (FLIP-156 and further work aim to address this [1]). At the moment, you can either - deploy the jobs in separate clusters (per-j

Does it support rate-limiting in flink 1.12?

When I run a sql job with blink planner in my cluster,the task is almost preemption the whole resources in the cluster, and this is a bad effect to the stream task.As it is not necessary on speed,so is there any way to control the rate in my batch task? this is the machine performance in ru