On Apr 27, 2016, at 2:57 PM, Gourav Rattihalli <[email protected]> wrote: > > > Hi Suresh, > > First of all I would like to commend you for starting this discussion. This > is exactly kind of the mailing list engagement we would like to see as part > of your GSOC. You already drew Emre’s attention, often discussions done in > isolation undermine this impact. So good job on a start and we will expect > over the summer you will engage lot more. > > > Thanks. Yes, I plan to actively use this mailing list to keep my design > focused on real use cases, and also to get regular feedback. > > Application dependency libraries, compiler versions can quickly get tangled. > You can choose to deploy Lmod (https://github.com/TACC/Lmod > <https://github.com/TACC/Lmod>) within a VM to alleviate this but that will > be an over kill. Eventually containers will hopefully alleviate this problem. > >> The worker VMs need to have the Mesos libraries installed and the Mesos >> master needs to be configured with the IPs of the worker VMs. So, if new VMs >> are created each time, we will develop the playbook for the creation of the >> Mesos cluster on the fly, and then launch the application on it. > > > Can the slave not register itself dynamically and report its IP Address? With > either multiple applications per VM or one per VM, you will need to look into > the capability of a persistent master and short lived elastic slave instances > right? May be I am mis-understating your questions. Can you help clarify this > more? > > > Yes, this is possible, and was the first of the two options I had emailed.
Can you elaborate on the second option, may be I missed what you tried to convey? Lets put aside how many applications per vm for this discussion since thats orthogonal. > To support this kind of elasticity, we will work on developing a > script/playbook to install the Mesos libraries on the new slaves/agents and > connect them with the master. Can you please develop them in Airavata repo itself? I mean lets not make perfection be the enemy here and start sending pull requests as you bake these playbooks. You can put the scripts here - https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks <https://github.com/apache/airavata/tree/develop/modules/cloud/ansible-playbooks> (note that it is develop branch). You can read about contributions here - http://airavata.staging.apache.org/community/how-to-contribute-code.html <http://airavata.staging.apache.org/community/how-to-contribute-code.html> (we need to move these instructions to the new website). Suresh > > Thanks, > Gourav > >> On Apr 27, 2016, at 1:21 PM, Emre Brookes <[email protected] >> <mailto:[email protected]>> wrote: >> >> Pierce, Marlon wrote: >>> Thanks, Gourav, this is great. How will the design differ in the two cases? >>> >>> Since installing the applications is not a simple task and there aren’t >>> good docker libraries (for example) for many of them, I would assume that >>> we would have a collection of VMs that have the codes already installed. >> FWIW, that model would work well with GenApp usage also. Although, I can >> imagine a scenario where a part of job initialization script (running under >> the already "code installed" VM) checks for updates to the code and installs >> them, but that can be independent of Airavata/Aurora/Mesos's knowledge. >> >> -Emre >> >>> >>> Marlon >>> >>> >>> From: Gourav Rattihalli <[email protected] >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>>> >>> Reply-To: "[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>" >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>> >>> Date: Wednesday, April 27, 2016 at 1:01 PM >>> To: "[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>" >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>> >>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream >>> >>> Hi Dev's, >>> >>> I have been working on the integration of Apache Aurora and Mesos with >>> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be >>> created using on Jetstream. I want to understand how Airavata is likely to >>> use the VMs for applications. Will airavata re-use existing VMs for >>> successive applications for a given community, or will new VMs be created >>> for each application? This will decide how we design the automated creation >>> of a Mesos cluster using the new VMs that are created. I understand that we >>> will have a Mesos master VM that will run all the time on Jetstream. >>> >>> Here is the Job submission flow that I am assuming: >>> >>> Airavata Client--> Airavata >>> Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will >>> create .aurora configuration file that will be used to launch the >>> job)-->Aurora/Mesos-->VMs >>> >>> So, I will design a module that will work with Orchestrator-->GFAC to >>> generate the appropriate Apache Aurora script to be submitted to the Aurora >>> master. >>> >>> Please let me know if there are any comments, suggestions on this plan. >>> >>> -- >>> Regards, >>> Gourav Rattihalli >> > > > > > -- > Regards, > Gourav Rattihalli
