Hi Song, Thank you for the clarification. Now I know TaskManagers are automatically allocated. Yet, I am still not very clear how the TMs are allocated. I'm guessing the allocation process would be: On the job side, I have a job with each operator parallelism=5. Since one TaskManager has one slot. It means this job would need 5 TaskManagers becase one pipeline would need one slot according to the task scheduling policy. On the Mesos side, let's say currently there are no available TaskManagers. Then Mesos would spawn 5 new TaskManagers for this job. Is it? If what I am guessing is right, then let's say we have 4 physical computing nodes for the Flink TaskManagers. How would Mesos place the 5 new TaskManagers into the 4 physical computing nodes? Is it juts a Round-Robin fashion? Kind regards Chase
On Fri, May 24, 2019 at 4:10 AM Xintong Song <tonysong...@gmail.com> wrote: > Hi black, > > If you are running Flink on Yarn or Mesos, Flink will automatically > allocate resource and launch new TaskManagers as needed. > > If you are using Flink standalone mode, then the easiest way is to enable > slot sharing and set all the vertices into the same group (which is by > default). In that way, the total slots (or number of TaskManagers if you > config on slot for each TaskManager) needed for running the job would be > the maximum parallelism of the job graph vertices. Further information on > slot sharing could be found here > <https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/runtime.html#task-slots-and-resources> > . > > Thank you~ > > Xintong Song > > > > On Thu, May 23, 2019 at 11:49 PM black chase <chaseandbl...@gmail.com> > wrote: > >> >> Hi, >> >> I am redesigning the scheduler of the JobManager to place tasks of a job >> across TaskManagers accroding to a scheduling policy. >> >> I am reading the Flip-6 proposal and found that the common case is "one >> TaskManager launchs one slot", and "one Flink cluster serves one job". But >> I did not find how many TaskManagers to launch in a computing node. Is >> there any common practice for this ? >> >> -- >> Best Regards! >> Pengcheng Duan >> > -- Best Regards!
