Re: Using Amazon EC2 Spot instances with Flink

My company is running Flink in Kubernetes with spot instances for both JM and TM. Feel free to reach out. On Thu, Mar 24, 2022, 20:33 Ber, Jeremy wrote: > > https://aws.amazon.com/blogs/compute/optimizing-apache-flink-on-amazon-eks-using-amazon-ec2-spot-instances/ > > > > Sharing this link FWIW

Re: Flink kafka consumer disconnection, application processing stays behind

Hi Isidoros, Your watermark strategy looks fine to me. I’m not quite sure if it is related. Best regards, Qingsheng > On Mar 24, 2022, at 21:11, Isidoros Ioannou wrote: > > Hi Qingsheng, > > thank you a lot for you response. > The message I see from the consumer before the log exception I

Flink Web ui not stable in kubernetes?

Hello all, i've been setting up flink in my kubernetes cluster with 2 job managers and 1 task manager (custom helm chart i wrote, no flink CLI used). I can access the web ui, but often it seems to switch pods which i am connected to and as soon as i am connected to the standby job manager it doesn

Where can I find flink-kubernetes-operator project?

Hello, I came across this document https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/ and would like to give it a try. But I have not been able to find it anywhere in Flink's main repository. Could someone please point me in the

flink cluster startup time

Hello, We run flink using the spotify flink Kubernetes operator (job cluster mode). Everything works fine, including upgrades and crash recovery. We do not run the job manager in HA mode. One of the problems we have is that upon upgrades (or during testing), the startup time of the flink clu

Re: Where can I find flink-kubernetes-operator project?

Hi Makas! The repo is here: https://github.com/apache/flink-kubernetes-operator We should add a link somewhere in the docs :) Cheers, Gyula On Fri, Mar 25, 2022 at 9:45 AM Makas Tzavellas wrote: > Hello, > > I came across this document > https://nightlies.apache.org/flink/flink-kubernetes-op

Re: Where can I find flink-kubernetes-operator project?

Thank you! On Fri, Mar 25, 2022 at 4:55 PM Gyula Fóra wrote: > Hi Makas! > > The repo is here: https://github.com/apache/flink-kubernetes-operator > > We should add a link somewhere in the docs :) > > Cheers, > Gyula > > > On Fri, Mar 25, 2022 at 9:45 AM Makas Tzavellas > wrote: > >> Hello, >>

Re: Flink Web ui not stable in kubernetes?

Hello Sebastian, Multiple versions of Flink 1.14.x are known to have issue with UI/CLI, please switch to Flink 1.14.4. Best regards, Guillaume On 25/03/2022 08.42, Sebastian Struss wrote: Hello all, i've been setting up flink in my kubernetes cluster with 2 job managers and 1 task manager

Re: Flink Web ui not stable in kubernetes?

Hi Guillaume, thank you for this great hint! It indeed fixed the mentioned issue. Just from reading the changelog of 1.14.4 i would not have known that this fix is included, maybe i was searching for the wrong stuff though. Have a great day! Sebastian On Fri, Mar 25, 2022 at 10:51 AM Guillaume V

Re: Flink Web ui not stable in kubernetes?

Hi, I agree that the changelog description of https://issues.apache.org/jira/browse/FLINK-25732 is talking only about technical root cause, not user consequences. I have added a comment in https://issues.apache.org/jira/browse/FLINK-25732. Have a nice day ! Guillaume On 25/03/2022 10.31, Se

"Native Kubernetes" sample in Flink documentation fails. JobManager Web Interface is wrongly generated. [Flink 1.14.4]

I am getting the following error when I try to execute sample at Flink documentation - Native Kubernetes. I have succedded to execute the first command in documentation by adding som

Re: DBT-flink profile?

Hi, use is perhaps not the right word (yet) rather experiment. But both would be relevant. And in particular, also the streaming option. I also just found: https://materialize.com/docs/guides/dbt/ outlining how dbt and streaming could potentially be married. Perhaps their integration could serve

flink-kubernetes-operator: Flink deployment stuck in scheduled state when increasing resource CPU above 1

Hi, I have been experimenting with flink-kubernetes-operator today and it's very cool. However, the Flink application will be stuck in scheduled state if I increase the CPU to anything above 1 for JobManager and TaskManager. It works fine if I keep the CPU to 1. I am testing the Flink deployment

Re: flink-kubernetes-operator: Flink deployment stuck in scheduled state when increasing resource CPU above 1

It's worth checking the deployment->replica set->pod chain for error message. On Fri, Mar 25, 2022, 19:49 Makas Tzavellas wrote: > Hi, > > I have been experimenting with flink-kubernetes-operator today and it's > very cool. However, the Flink application will be stuck in scheduled state > if I i

Re: "Native Kubernetes" sample in Flink documentation fails. JobManager Web Interface is wrongly generated. [Flink 1.14.4]

The root cause might be the LoadBalancer could not really work in your environment. We already have a ticket to track this[1] and will try to get it resolved in the next release. For now, could you please have a try by adding "-Dkubernetes.rest-service.exposed.type=NodePort" to your session and su

Re: flink-kubernetes-operator: Flink deployment stuck in scheduled state when increasing resource CPU above 1

Could you please share the result of "kubectl describe pods" when getting stuck? It will be very useful to help to figure out the root cause. I guess it might be related to insufficient resources for minikube. Best, Yang Őrhidi Mátyás 于2022年3月26日周六 03:12写道： > It's worth checking the deploymen

Re: flink-kubernetes-operator: Flink deployment stuck in scheduled state when increasing resource CPU above 1

Thanks everyone for the responses. I have checked and noticed that the pod fails to start due to insufficient resources with kubectl describe pods. It is able to run only if I set jobManager to 1 CPU and taskManager to 1.5 CPU at maximum. But it's good to know that it wasn't due to an incorrect c

Re: Flatmap operator in an Asynchronous call

I'm not able to use asyncIO because the file will not fit in memory. I thought that flatmap will allow me to enrich/process records while downloading instead of waiting for the whole file to get downloaded. The solution works but its not scalable because i'm not able to use AsynFunction in Flatmap.