Hi Matthias, Thanks for your reply. In my case, yes the upstream operator for the operator which is not distributed evenly among task managers is a flink Kafka connector with a rebalance(shuffling).
Regards, Vignesh On Tue, 23 Mar, 2021, 6:48 pm Matthias Pohl, <matth...@ververica.com> wrote: > There was a similar discussion recently in this mailing list about > distributing the work onto different TaskManagers. One finding Xintong > shared there [1] was that the parameter cluster.evenly-spread-out-slots is > used to evenly allocate slots among TaskManagers but not how the tasks are > actually distributed among the allocated slots. It would be interesting to > know more about your job. If the upstream operator does some shuffling, you > might run into the issue of the task executions not being distributed > evenly anymore. > > Matthias > > [1] > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Evenly-Spreading-Out-Source-Tasks-tp42108p42235.html > > On Tue, Mar 23, 2021 at 1:42 PM Matthias Pohl <matth...@ververica.com> > wrote: > >> Hi Vignesh, >> are you trying to achieve an even distribution of tasks for this one >> operator that has the parallelism set to 16? Or do you observe the >> described behavior also on a job level? >> I'm adding Chesnay to the thread as he might have more insights on this >> topic. >> >> Best, >> Matthias >> >> On Mon, Mar 22, 2021 at 6:31 PM Vignesh Ramesh <vicki.ram...@gmail.com> >> wrote: >> >>> Hello Everyone, >>> >>> Can someone help me with a solution? >>> >>> I have a flink job(2 task-managers) with a job parallelism of 64 and >>> task slot of 64. >>> I have a parallelism set for one of the operators as 16. This >>> operator(16 parallelism) slots are not getting evenly distributed across >>> two task managers. It often takes higher task slots like 10/11 in one task >>> manager and 5/6 in other task manager. >>> >>> I'am using flink version 1.11.2. I tried adding >>> cluster.evenly-spread-out-slots: >>> true but it didn't work. Any solution is greatly appreciated. >>> >>> Thanks in advance, >>> >>> Regards, >>> Vignesh >>> >>>
