I am hoping to get some advice on how to build out my framework scheduler for a job scheduling/processing platform, so I don't paint myself in a corner with any potential limitations in mesos resource allocation or the java SchedulerDriver capabilities. Some background:
- I have a JVM based job scheduling engine that schedules jobs based on repeating time patterns (like cron) and also allows for on-demand job execution. - The system supports users creating an number of partitions/queues that have resource preferences defined by the user. So one partitions may run a certain set of jobs and another partition may run another set of jobs. Currently the resources are statically allocated to one or more computers and the user can define the max number of jobs that can be run on each host in the Partition. - The scheduling engine is all run out of a single JVM process that is heavily threaded/concurrent. So my initial thinking is to create a mesos Sched/Framework instance per partition/queue thread in the my engine (there could be many partitions since this is user created/controlled via GUI). Each SchedulerDriver (living in the JVM) would request some set of resources from Mesos (as defined by the size of of the queue of jobs running in the Partition) when jobs become ready to run on reoccurring time schedules. Now my question is, if it is practical to have multiple mesos Schedulers Frameworks in a single JVM and each with different resource requirements. Or should I build a single central mesos scheduler in the JVM and have all my partitions/queues make request to this central mesos Scheduler that talks to the mesos master? I am not sure how mesos scales if for example you have many framework schedulers running out of the same JVM/process? And in general if you have a large number of Scheduler instances will this cause resource distribution problems with on one scheduler getting too many resources allocated....etc. So the bottom line question is should have create/instantiate a single central Framework Scheduler in my engine (that proxies requests/offers with mesos) or can I create one per Partition thread. The per partition thread approach can give me finer resource control (request, Filters,....etc) but I am not sure if this is practical. Thanks, Sam Taha http://www.grandlogic.com
