Re: Uneven TM Distribution of Flink on YARN

CCing @Weihua Hu , who is an expert on this. Do you have any ideas on the phenomenon here? Best, Zhanghao Chen From: Lu Niu Sent: Tuesday, August 29, 2023 12:11:35 PM To: Chen Zhanghao Cc: Kenan Kılıçtepe ; user Subject: Re: Uneve

Re: Uneven TM Distribution of Flink on YARN

Thanks for your reply. The interesting fact is that we also managed spark on yarn. However. Only the flink cluster is having the issue. I am wondering whether there is a difference in the implementation on flink side. Best Lu On Mon, Aug 28, 2023 at 8:38 PM Chen Zhanghao wrote: > Hi Lu Niu, >

回复: Uneven TM Distribution of Flink on YARN

Hi Lu Niu, TM distribution on YARN nodes is managed by YARN RM and is out of the scope of Flink. On the other hand, cluster.evenly-spread-out-slots forces even distribution of tasks among Flink TMs, and has nothing to do with your concerns. Also, the config currently only supports Standalone mo

Re: Table API table2datastream (toChangelogStream)

Hi Ravi, Maybe it's due to *execution.batch.adaptive.auto-parallelism.enabled* Option. How about trying to configure *execution.batch.adaptive.auto-parallelism.enabled* to *false* and apply other parallelism. Best, Dongwoo 2023년 8월 27일 (일) 오후 8:00, ravi_suryavanshi.yahoo.com via user < user@flink

Re: Uneven TM Distribution of Flink on YARN

Thanks for the reply. We've already set cluster.evenly-spread-out-slots = true Best Lu On Mon, Aug 28, 2023 at 1:23 PM Kenan Kılıçtepe wrote: > Have you checked config param cluster.evenly-spread-out-slots ? > > > On Mon, Aug 28, 2023 at 10:31 PM Lu Niu wrote: > >> Hi, Flink users >> >> We hav

Re: Task Manager getting killed while executing sql queries.

Can it be a memory leak? Have you observed the memory consumption of task managers? Once, task manager crush issue happened for me and it was OOM. On Mon, Aug 28, 2023 at 9:12 PM Neha Rawat wrote: > Hi, > > > > Need some help with the below situation. If would be great if someone > could give s

Re: Uneven TM Distribution of Flink on YARN

Have you checked config param cluster.evenly-spread-out-slots ? On Mon, Aug 28, 2023 at 10:31 PM Lu Niu wrote: > Hi, Flink users > > We have recently observed that the allocation of Flink TaskManagers in our > YARN cluster is not evenly distributed. We would like to hear your thoughts > on this

Registration open for Community Over Code North America

Hello! Registration is still open for the upcoming Community Over Code NA event in Halifax, NS! We invite you to register for the event https://communityovercode.org/registration/ Apache Committers, note that you have a special discounted rate for the conference at US$250. To take advantage of th

Uneven TM Distribution of Flink on YARN

Hi, Flink users We have recently observed that the allocation of Flink TaskManagers in our YARN cluster is not evenly distributed. We would like to hear your thoughts on this matter. 1. Our setup includes Flink version 1.15.1 and Hadoop 2.10.0. 2. The uneven distribution is that out of a 370-node

Custom source function in PyFlink

Hey folks, I'm looking for an example for creating a custom source in PyFlink. The one that I found in the tests is a wrapper around a java class: def test_add_custom_source(self): custom_source = SourceFunction( "org.apache.flink.python.util.MyCustomSourceFunction") ds = self.env.add_source(cust

Checkpoint/savepoint _metadata

Hi team, We use flink 1.16.0 with openjdk-11-jre mainly to run streaming jobs. We do checkpoints with 2 min interval and savepoint when deploying new job version. We also use rocksdb state backend for most of them. We had a streaming job running for long without any issue and during a new deploym