I think currently flink doesn't support your case, and another idea is that you can set the parallelism of all operators to 64, then it will be evenly distributed to the two taskmanagers.
Vignesh Ramesh <vicki.ram...@gmail.com> 于2021年3月25日周四 上午1:05写道： > Hi Matthias, > > Thanks for your reply. In my case, yes the upstream operator for the > operator which is not distributed evenly among task managers is a flink > Kafka connector with a rebalance(shuffling). > > Regards, > Vignesh > > On Tue, 23 Mar, 2021, 6:48 pm Matthias Pohl, <matth...@ververica.com> > wrote: > >> There was a similar discussion recently in this mailing list about >> distributing the work onto different TaskManagers. One finding Xintong >> shared there [1] was that the parameter cluster.evenly-spread-out-slots is >> used to evenly allocate slots among TaskManagers but not how the tasks are >> actually distributed among the allocated slots. It would be interesting to >> know more about your job. If the upstream operator does some shuffling, you >> might run into the issue of the task executions not being distributed >> evenly anymore. >> >> Matthias >> >> [1] >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Evenly-Spreading-Out-Source-Tasks-tp42108p42235.html >> >> On Tue, Mar 23, 2021 at 1:42 PM Matthias Pohl <matth...@ververica.com> >> wrote: >> >>> Hi Vignesh, >>> are you trying to achieve an even distribution of tasks for this one >>> operator that has the parallelism set to 16? Or do you observe the >>> described behavior also on a job level? >>> I'm adding Chesnay to the thread as he might have more insights on this >>> topic. >>> >>> Best, >>> Matthias >>> >>> On Mon, Mar 22, 2021 at 6:31 PM Vignesh Ramesh <vicki.ram...@gmail.com> >>> wrote: >>> >>>> Hello Everyone, >>>> >>>> Can someone help me with a solution? >>>> >>>> I have a flink job(2 task-managers) with a job parallelism of 64 and >>>> task slot of 64. >>>> I have a parallelism set for one of the operators as 16. This >>>> operator(16 parallelism) slots are not getting evenly distributed across >>>> two task managers. It often takes higher task slots like 10/11 in one task >>>> manager and 5/6 in other task manager. >>>> >>>> I'am using flink version 1.11.2. I tried adding >>>> cluster.evenly-spread-out-slots: >>>> true but it didn't work. Any solution is greatly appreciated. >>>> >>>> Thanks in advance, >>>> >>>> Regards, >>>> Vignesh >>>> >>>>
