Re: Metascheduler work
Hi Dimuthu, > On Dec 17, 2018, at 9:55 AM, DImuthu Upeksha > wrote: > > Metascheduling Usecase 3: Users within a gateway need to fairly use a > community account. Computational resources like XSEDE enforce fair-share > across users, but since gateway job submissions are funneled through a single > community account, different users within a gateway are impacted. Airavata > will need to implement fair-share scheduling among these users to ensure fair > use of allocations as well as allowable queue limits and work with resources > policies. > > Comments > 1. Is there any existing work regarding this usecase? I can remember that two > students created a UI to enforce these limits and save in a database. > I'm guessing that the fair share regarding metascheduling is a little different from the allocation manager work done by IU students in the spring. Here fair share would mean fairly scheduling jobs so that no one user gets a greater share of running jobs. For example, if a queue only allows 50 jobs per user and all jobs from a gateway are submitted as a single community account user, and you have gateway user A submit 50 jobs and gateway users B and C submit 1 job each, then you would want to avoid the situation where all 50 of user A's jobs get submitted and user B and C's jobs are waiting on them to complete before their jobs get submitted to the queue. You would rather want to have user B and user C's jobs get submitted and 48 of user A's jobs. smime.p7s Description: S/MIME cryptographic signature
Re: Metascheduler work
The solution I provided was to run MPI tasks with Mesos was with the help of Docker Swarm. It required me to develop a Mesos Framework and an Executor [here <http://mesos.apache.org/documentation/latest/app-framework-development-guide/>]. Mesos containers could not communicate with each other and that was the reason to bring Docker Swarm into the picture. However now, Mesos container can communicate with help of Calico [here <https://docs.projectcalico.org/v1.5/getting-started/mesos/>]. Calico provides an overlay network and each Mesos container gets a unique IP address. Thanks Pankaj On Mon, Dec 17, 2018 at 12:13 PM Pamidighantam, Sudhakar V < spami...@illinois.edu> wrote: > I thought Pankaj presented a way to implement MPI executions through mesos > more recently. > > He may want to comment on this.. > > > > Thanks, > > Sudakar. > > > > *From: *Marlon Pierce > *Reply-To: * > *Date: *Monday, December 17, 2018 at 8:52 AM > *To: *"dev@airavata.apache.org" > *Subject: *Re: Metascheduler work > > > > Hi Dimuthu, > > > > This is something we should re-evaluate. Mangirish Wangle looked at Mesos > integration with Airavata back in 2016, but he ultimately ran into many > difficulties, including getting MPI jobs to work, if I recall correctly. > > > > Marlon > > > > > > *From: *"dimuthu.upeks...@gmail.com" > *Reply-To: *dev > *Date: *Sunday, December 16, 2018 at 7:30 AM > *To: *dev > *Subject: *Metascheduler work > > > > Hi Folks, > > > > I found this [1] mail thread and the JIRA ticket [2] which have discussed > about coming up with an Airavata specific job scheduler. At the end of the > discussion, seems like an approach based on Mesos has been chosen to > tryout. Is there any other discussion/ documents regarding this topic? Has > anyone worked on this and if so, where are the code / design documents? > > > > [1] > https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results > > [2] https://issues.apache.org/jira/browse/AIRAVATA-1436 > > > > Thanks > > Dimuthu >
Re: Metascheduler work
I thought Pankaj presented a way to implement MPI executions through mesos more recently. He may want to comment on this.. Thanks, Sudakar. From: Marlon Pierce Reply-To: Date: Monday, December 17, 2018 at 8:52 AM To: "dev@airavata.apache.org" Subject: Re: Metascheduler work Hi Dimuthu, This is something we should re-evaluate. Mangirish Wangle looked at Mesos integration with Airavata back in 2016, but he ultimately ran into many difficulties, including getting MPI jobs to work, if I recall correctly. Marlon From: "dimuthu.upeks...@gmail.com" Reply-To: dev Date: Sunday, December 16, 2018 at 7:30 AM To: dev Subject: Metascheduler work Hi Folks, I found this [1] mail thread and the JIRA ticket [2] which have discussed about coming up with an Airavata specific job scheduler. At the end of the discussion, seems like an approach based on Mesos has been chosen to tryout. Is there any other discussion/ documents regarding this topic? Has anyone worked on this and if so, where are the code / design documents? [1] https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results [2] https://issues.apache.org/jira/browse/AIRAVATA-1436 Thanks Dimuthu
Re: Metascheduler work
Hi Marlon, Thanks for the clarification. Yes I agree that we need to re-evaluate both the use case and the approach we are going to implement this. Hi All, Considering current requirements of gateways, we somehow need a better job throttling/ scheduling proxy before the actual schedulers. I will simply copy and paste the existing use cases mentioned in the wiki and please share your views on the validity of these use cases in current gateways. Once we have come up with a concrete set of requirements, then let's think about the technical side of the solution. I have added my point of view on each usecase in comments section. *Metascheduling Usecase 1: Users/Gateways submitting a series of jobs to a resource. Whereas a resource enforces a per user job limit within a queue and ensure fair use of the clusters ((example: stampede allows 50 jobs per user in the normal queue). Airavata will need to implement queues and throttle jobs respecting the max-job-per-queue limits of a underlying resource queue. * Comments: 1. What will happen to the pending jobs in the Airavata queue? Will they stay in the queue forever until the queues in the clusters become available or is there an expiry? 2. Are we assuming that Airavata is the only entry point for the queues in the clusters or can there be some other gateways/ independent users submitting jobs to the same queue? *Metascheduling Usecase 2: Users/Gateways delegate job scheduling across available computational resources to Airavata. Airavata will need to implement schedulers which become aware of existing loads on the clusters and spread jobs efficiently. The scheduler should also be able to get access to heuristics on previous executions and current requirements which includes job size (number of nodes/cores), memory requirements, wall time estimates and so forth. * Comments: 1. Even though the first part makes sense for me, second part looks more like a machine learning problem and a nice to have a feature. Is this something like user comes into portal and launch an application with a set of inputs and Airavata decides in which machine / cluster should this job run? *Metascheduling Usecase 3: Users within a gateway need to fairly use a community account. Computational resources like XSEDE enforce fair-share across users, but since gateway job submissions are funneled through a single community account, different users within a gateway are impacted. Airavata will need to implement fair-share scheduling among these users to ensure fair use of allocations as well as allowable queue limits and work with resources policies.* Comments 1. Is there any existing work regarding this usecase? I can remember that two students created a UI to enforce these limits and save in a database. Thanks Dimuthu On Mon, Dec 17, 2018 at 7:22 PM Pierce, Marlon wrote: > Hi Dimuthu, > > > > This is something we should re-evaluate. Mangirish Wangle looked at Mesos > integration with Airavata back in 2016, but he ultimately ran into many > difficulties, including getting MPI jobs to work, if I recall correctly. > > > > Marlon > > > > > > *From: *"dimuthu.upeks...@gmail.com" > *Reply-To: *dev > *Date: *Sunday, December 16, 2018 at 7:30 AM > *To: *dev > *Subject: *Metascheduler work > > > > Hi Folks, > > > > I found this [1] mail thread and the JIRA ticket [2] which have discussed > about coming up with an Airavata specific job scheduler. At the end of the > discussion, seems like an approach based on Mesos has been chosen to > tryout. Is there any other discussion/ documents regarding this topic? Has > anyone worked on this and if so, where are the code / design documents? > > > > [1] > https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results > > [2] https://issues.apache.org/jira/browse/AIRAVATA-1436 > > > > Thanks > > Dimuthu >
Re: Metascheduler work
Hi Dimuthu, This is something we should re-evaluate. Mangirish Wangle looked at Mesos integration with Airavata back in 2016, but he ultimately ran into many difficulties, including getting MPI jobs to work, if I recall correctly. Marlon From: "dimuthu.upeks...@gmail.com" Reply-To: dev Date: Sunday, December 16, 2018 at 7:30 AM To: dev Subject: Metascheduler work Hi Folks, I found this [1] mail thread and the JIRA ticket [2] which have discussed about coming up with an Airavata specific job scheduler. At the end of the discussion, seems like an approach based on Mesos has been chosen to tryout. Is there any other discussion/ documents regarding this topic? Has anyone worked on this and if so, where are the code / design documents? [1] https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results [2] https://issues.apache.org/jira/browse/AIRAVATA-1436 Thanks Dimuthu smime.p7s Description: S/MIME cryptographic signature
Metascheduler work
Hi Folks, I found this [1] mail thread and the JIRA ticket [2] which have discussed about coming up with an Airavata specific job scheduler. At the end of the discussion, seems like an approach based on Mesos has been chosen to tryout. Is there any other discussion/ documents regarding this topic? Has anyone worked on this and if so, where are the code / design documents? [1] https://markmail.org/message/tdae5y3togyq4duv#query:+page:1+mid:tdae5y3togyq4duv+state:results [2] https://issues.apache.org/jira/browse/AIRAVATA-1436 Thanks Dimuthu