How to specify both the resource limit and resource request for JM/TM in flink-operator

  Hi, community. I wonder how can i specify both the resource request and limit for JM/TM in the podTemplate using flink-operator? We have the need to set the request resource and limit resource to different value. For example: jobManager: limits: cpu: 500m memory: 500Mi reques

回复: How to specify both the resource limit and resource request for JM/TM in flink-operator

Seems i can achieve this by specify the "kubernetes.jobmanager.cpu.limit-factor" and "kubernetes.taskmanager.cpu.limit-factor" in flink properties. Those parameter are supported since flink 1.15 发件人: Liting Liu (litiliu) 发送时间: 2023年1月12日 16:28 收件人: user 主题: H

Re: Unable to list jobs in flink cluster with multiple jobManagers

I think it is a bug: https://issues.apache.org/jira/browse/FLINK-25732 Yael Adsl 于2022年12月12日周一 23:56写道： > > Hi, > > We are running a flink cluster (Flink version 1.14.3) on kubernetes with > high-availablity.type: kubernetes. We have 3 jobmanagers. When we send jobs > to the flink cluster, we

Re: Flink reactive mode for application clusters on AWS EKS

The adaptive scheduler and reactive mode both already support application clusters since 1.13. https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/ On 19/12/2022 10:17, Tamir Sagi wrote: Hey, We are running stream jobs on application clusters (v1.15.2) o

Re: Using filesystem plugin with MiniCluster

There is no good way in 1.15 IIRC. Adding a dependency on flink-s3-fs-hadoop _can_ work, if you dont run into dependency conflicts. Otherwise you have to create a plugin manager yourself, point it to some local directory via a system property (I think?), and then eagerly call FileSystem#init

Re: Async IO & Retry: how to get error details

Retry logic and per-request timeouts should be setup within asyncInvoke() (with all error-handling being done via plain CompletableFuture logic), with timeout() sort of acting as a global timeout after which you want the job to fail (e.g., to guard against mistakes in the asyncInvoke() logic).

Re: Metrics exposed by flink containing long label values

See https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/#scope-variables-excludes On 06/01/2023 12:09, Surendra Lalwani via user wrote: Hi Team, We are exposing metrics by Flink to prometheus but we are seeing it contains various labels even including o

Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay, Thank you for your response. Since we are running our Flink jobs on EKS (Elastic Kubernetes Service) I was asking regarding Application cluster on Kubernetes. The documentations I referred to clearly state that it is not supported, the same as shown on Flink website. [cid:29a054

Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay, Just to be more clear, I'm talking about plans to support reactive mode for application clusters in Native Kubernetes. https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode Thanks, Tamir. ___

Re: Flink reactive mode for application clusters on AWS EKS

Hey! I think the reactive scaling is a somewhat misunderstood feature. It only works in standalone deployments (not in Kubernetes native for instace) and it doesn't actually provide any autoscaling functionality on its own. You would have to implement your scaling logic yourself somehow (Kubernete

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hi @Gyula Fóra, Does this mean, with Kubernetes operator, we can have reactive mode in native flink which is in app mode ? [ Not just standalone app mode ] Regards, Swathi C From: Gyula Fóra Sent: Thursday, January 12, 2023 11:14 PM To: Tamir Sagi Cc: Chesnay Sche

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

No but the Kubernetes operator itself already provides similar feature set. Not sure why you want the reactive mode in the first place . If it's because you want to implement auto scaling on top of it, then I think the operator is a better alternative. I think you should try to understand what ex

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Got it, so this means, we should have standalone app mode cluster which is managed by a flink Kubernetes operator and the operator would update the replicas based on the metrics ( autoscale ) which in-tern changes the parallelism as reactivemode is enabled. Regards, Swathi C From: Gyula Fóra

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

What I am trying to say is use the Kubernetes operator with Native (default) mode and forget about reactive . The operator does everything you wwant plus has an actual autoscaler. Gyula On Fri, 13 Jan 2023 at 07:24, Swathi Chandrashekar wrote: > Got it, so this means, we should have standalone

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Got it, thanks @Gyula Fóra 😊 From: Gyula Fóra Sent: Friday, January 13, 2023 12:09 PM To: Swathi Chandrashekar Cc: Chesnay Schepler ; Tamir Sagi ; user@flink.apache.org Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS What I am t

HybridSource: how to specify different watermark strategies for underlying sources

Hello, I'm trying to use the HybridSource to read from a bounded FLIP-27 Iceberg source and an unbounded FLIP-27 PubSub source. Based on my searches, the most common patterns for assigning timestamps and watermarks with FLIP-27 sources are either via the fromSource method in StreamingExecutionEnvi