There was a similar discussion recently in this mailing list about distributing the work onto different TaskManagers. One finding Xintong shared there [1] was that the parameter cluster.evenly-spread-out-slots is used to evenly allocate slots among TaskManagers but not how the tasks are actually distributed among the allocated slots. It would be interesting to know more about your job. If the upstream operator does some shuffling, you might run into the issue of the task executions not being distributed evenly anymore.
Matthias [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Evenly-Spreading-Out-Source-Tasks-tp42108p42235.html On Tue, Mar 23, 2021 at 1:42 PM Matthias Pohl <matth...@ververica.com> wrote: > Hi Vignesh, > are you trying to achieve an even distribution of tasks for this one > operator that has the parallelism set to 16? Or do you observe the > described behavior also on a job level? > I'm adding Chesnay to the thread as he might have more insights on this > topic. > > Best, > Matthias > > On Mon, Mar 22, 2021 at 6:31 PM Vignesh Ramesh <vicki.ram...@gmail.com> > wrote: > >> Hello Everyone, >> >> Can someone help me with a solution? >> >> I have a flink job(2 task-managers) with a job parallelism of 64 and task >> slot of 64. >> I have a parallelism set for one of the operators as 16. This operator(16 >> parallelism) slots are not getting evenly distributed across two task >> managers. It often takes higher task slots like 10/11 in one task manager >> and 5/6 in other task manager. >> >> I'am using flink version 1.11.2. I tried adding >> cluster.evenly-spread-out-slots: >> true but it didn't work. Any solution is greatly appreciated. >> >> Thanks in advance, >> >> Regards, >> Vignesh >> >>
