Well, it depends on how many resource are needed for one pipeline of you job and how many resource are are configured for each TaskExecutor. In addition, the resource of each TaskManager also depends on the job's resource needs and your environment. So having one slot for each TaskManager would be a simple choice because it avoids tuning these two relevant factors at the same time.
Thank you~ Xintong Song On Sat, May 25, 2019 at 4:54 AM black chase <chaseandbl...@gmail.com> wrote: > Hi Song, > You said "In that way, the total slots (or number of TaskManagers if you > config on slot for each TaskManager)", do you imply that one taskmanager > contains one slot? > Do you have some experience on how many slots to spawn for one > TaskManager? > I read the Flip-6, it says "For the sake of simplicity, the following > talks about “slots”, but one can think simply of “TaskManager” instead, > for the common case of a one-slot TaskManager.". > It seems the common practice is to have one slot for one taskmanager. > > Best, > Chase > > > On Fri, May 24, 2019 at 11:28 AM black chase <chaseandbl...@gmail.com> > wrote: > >> Yes true. I am trying to figure out how the TaskManagers are distributed >> across physical machines by Mesos and YARN. Maybe I shoud start a new >> thread for help. >> Thank you Song >> Best, >> Pengcheng >> >> On Fri, May 24, 2019 at 10:52 AM Xintong Song <tonysong...@gmail.com> >> wrote: >> >>> As far as I know, Flink does not have any requirements on how the >>> TaskManagers are distributed across physical machines. So I think it really >>> depends on the scheduling policy of the Mesos cluster. I'm not an expert on >>> Mesos, so correct me if I was wrong. >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> >>> On Fri, May 24, 2019 at 4:18 PM black chase <chaseandbl...@gmail.com> >>> wrote: >>> >>>> Hi Song, >>>> Thank you for the clarification. >>>> Now I know TaskManagers are automatically allocated. Yet, I am still >>>> not very clear how the TMs are allocated. >>>> I'm guessing the allocation process would be: >>>> On the job side, I have a job with each operator parallelism=5. Since >>>> one TaskManager has one slot. It means this job would need 5 TaskManagers >>>> becase one pipeline would need one slot according to the task scheduling >>>> policy. >>>> On the Mesos side, let's say currently there are no available >>>> TaskManagers. Then Mesos would spawn 5 new TaskManagers for this job. Is >>>> it? >>>> If what I am guessing is right, then let's say we have 4 physical >>>> computing nodes for the Flink TaskManagers. How would Mesos place the 5 new >>>> TaskManagers into the 4 physical computing nodes? Is it juts a Round-Robin >>>> fashion? >>>> Kind regards >>>> Chase >>>> >>>> >>>> >>>> On Fri, May 24, 2019 at 4:10 AM Xintong Song <tonysong...@gmail.com> >>>> wrote: >>>> >>>>> Hi black, >>>>> >>>>> If you are running Flink on Yarn or Mesos, Flink will automatically >>>>> allocate resource and launch new TaskManagers as needed. >>>>> >>>>> If you are using Flink standalone mode, then the easiest way is to >>>>> enable slot sharing and set all the vertices into the same group (which is >>>>> by default). In that way, the total slots (or number of TaskManagers if >>>>> you >>>>> config on slot for each TaskManager) needed for running the job would be >>>>> the maximum parallelism of the job graph vertices. Further information on >>>>> slot sharing could be found here >>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/runtime.html#task-slots-and-resources> >>>>> . >>>>> >>>>> Thank you~ >>>>> >>>>> Xintong Song >>>>> >>>>> >>>>> >>>>> On Thu, May 23, 2019 at 11:49 PM black chase <chaseandbl...@gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am redesigning the scheduler of the JobManager to place tasks of a >>>>>> job across TaskManagers accroding to a scheduling policy. >>>>>> >>>>>> I am reading the Flip-6 proposal and found that the common case is >>>>>> "one TaskManager launchs one slot", and "one Flink cluster serves one >>>>>> job". >>>>>> But I did not find how many TaskManagers to launch in a computing node. >>>>>> Is >>>>>> there any common practice for this ? >>>>>> >>>>>> -- >>>>>> Best Regards! >>>>>> Pengcheng Duan >>>>>> >>>>> >>>> >>>> -- >>>> Best Regards! >>>> >>> >> >> -- >> Best Regards! >> > > > -- > Best Regards! >
