Hi Mangirish, Yes now I noticed the scaling within the heat section. Yes it makes sense to leave it behind the orchestration layer not to re-invent that logic.
Airavata Orchestrator will be the natural plan to call the provisioning service and bootstrap the mesos cluster. The ansible I referred to are not yet contributed into the repo. I am cc’ing Pankaj and Renan who can probably make that contribution. You can read about their effort in http://onlinelibrary.wiley.com/doi/10.1002/cpe.3708/full <http://onlinelibrary.wiley.com/doi/10.1002/cpe.3708/full> Renan, Mangirish is proposing a project to programmatically interact with Cloud Interfaces (like Open Stack on Jetstream) and provision resources. I would assume then the component you have developed will take over and bootstrap the mesos cluster which GFac can then submit jobs to (through Aurora). Suresh > On Mar 24, 2016, at 9:14 PM, Mangirish Wagle <[email protected]> wrote: > > Hello, > > I was trying to understand the end result flow of the Airavata with Cloud > Orchestrator and had the following question:- > > Once the cluster has been setup, as we discussed, an ansible or some > configuration management tool would boostrap and configure mesos. Which > component in Airavata would host and call the ansible script and what event > would trigger it? > > Thanks. > > Regards, > Mangirish > > On Thu, Mar 24, 2016 at 9:07 PM, Mangirish Wagle <[email protected] > <mailto:[email protected]>> wrote: > Thanks for your feedback Suresh! > > I have mentioned about the Autoscaling in the Heat Orchestration solution, > which does the dynamic scaling of resources in an existing cloud. Please let > me know if you think that needs to be restructured. > > Also, I have updated the Google doc and Wiki with the revised proposal, after > making changes as per Marlon's review comments. > > I request you to please review again and check if there is anything that > needs still needs to be revised. > > Thank you! > > Regards, > Mangirish > > On Thu, Mar 24, 2016 at 7:18 PM, Suresh Marru <[email protected] > <mailto:[email protected]>> wrote: > Hi Mangirish, > > Your proposal has all the required good detail. One optional addition you can > clarify on if you can expand or contract resources to a previously > provisioned cloud. > > Suresh > >> On Mar 23, 2016, at 9:10 PM, Mangirish Wagle <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thanks Shameera for the info and sharing the JIRA Epic details. >> >> I have drafted my GSOC Proposal for the project and I request you to please >> review the same:- >> >> https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata >> >> <https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata> >> >> I shall submit this on the GSOC portal by tomorrow, once I get my enrollment >> verification proof. >> >> Regards, >> Mangirish >> >> >> >> On Wed, Mar 23, 2016 at 12:29 PM, Shameera Rathnayaka >> <[email protected] <mailto:[email protected]>> wrote: >> Hi Mangirish, >> >> Yes your above understanding is right. Gfac is like task executor which >> execute what ever task given by Orchestrator. >> >> Here is the epic https://issues.apache.org/jira/browse/AIRAVATA-1924 >> <https://issues.apache.org/jira/browse/AIRAVATA-1924>, Open stack >> integration is part of this epic, you can create a new top level jira ticket >> and create subtask under that ticket. >> >> Regards, >> Shameera. >> >> On Wed, Mar 23, 2016 at 12:20 PM Mangirish Wagle <[email protected] >> <mailto:[email protected]>> wrote: >> Thanks Marlon for the info. So what I get is that the Orchestrator would >> decide if the job needs to be submitted to cloud based cluster and route it >> to GFAC which would have a separate interfacing with the cloud cluster >> service. >> >> Also I wanted to know if there is any Story/ Epic created in JIRA for this >> project which I can use to create and track tasks? If not can I create one? >> >> Thanks. >> >> Regards, >> Mangirish >> >> On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon <[email protected] >> <mailto:[email protected]>> wrote: >> The Application Factory component is called “gfac” in the code base. This >> is the part that handles the interfacing to the remote resource (most often >> by ssh but other providers exist). The Orchestrator routes jobs to GFAC >> instances. >> >> From: Mangirish Wagle <[email protected] >> <mailto:[email protected]>> >> Reply-To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Date: Wednesday, March 23, 2016 at 11:56 AM >> To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata >> >> Hello Team, >> >> I was drafting the GSOC proposal and I just had a quick question about the >> integration of the project with Apache Airavata. >> >> Which is the component in Airavata that would call the service to provision >> the cloud cluster? >> >> I am looking at the Airavata architecture diagram and my understanding is >> that this would be treated as a new Application and would have a separate >> application interface in 'Application Factory' component. Also the workflow >> orchestrator would be having the intelligence to figure out which jobs to be >> submitted to cloud based clusters. >> >> Please let me know whether my understanding is correct. >> >> Thank you. >> >> Best Regards, >> Mangirish Wagle >> >> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Mangirish, please add your proposal to the GSOC 2016 site. >> >> From: Mangirish Wagle <[email protected] >> <mailto:[email protected]>> >> Reply-To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Date: Thursday, March 17, 2016 at 3:35 PM >> To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata >> >> Hello Dev Team, >> >> I had the opportunity to interact with Suresh and Shameera wherein we >> discussed an open requirement in Airavata to be addressed. The requirement >> is to expand the capabilities of Apache Airavata to submit jobs to cloud >> based clusters in addition to HPC/ HTC clusters. >> >> The idea is to dynamically provision a cloud cluster in an environment like >> Jetstream, based on the configuration figured out by Airavata, which would >> be operated by a distributed system management software like Mesos. An >> initial high level goals would be:- >> Airavata categorizes certain jobs to be run on cloud based clusters and >> figure out the required hardware config for the cluster. >> The proposed service would provision the cluster with the required resources. >> An ansible script would configure a Mesos cluster with the resources >> provisioned. >> Airavata submits the job to the Mesos cluster. >> Mesos then figures out the efficient resource allocation within the cluster >> and runs the job and fetches the result. >> The cluster is then deprovisioned automatically when not in use. >> The project would mainly focus on point 2 and 6 above. >> >> To start with, I am currently trying to get a working prototype of setting >> up compute nodes on an openstack environment using JClouds (Targetted for >> Jetstream). Also, I am planning to explore the option of using Openstack >> Heat engine to orchestrate the cluster. However, going ahead Airavata would >> be supporting other clouds like Amazon EC2 or Comet cluster, so we need to >> have a generic solution for achieving the goal. >> >> Another approach which might be efficient in terms of performance and time >> is using a container based clouds using Docker, Kubernetes which would have >> substantially less bootstrap time compared to cloud VMs. This would be a >> future prospect as we may not have all the clusters supporting >> containerization. >> >> This has been considered as a potential GSOC project and I would be working >> on drafting a proposal on this idea. >> >> Any inputs/ comments/ suggestions would be very helpful. >> >> Best Regards, >> Mangirish Wagle >> >> >> -- >> Shameera Rathnayaka >> > > >
