Hi Gourav, First of all I would like to commend you for starting this discussion. This is exactly kind of the mailing list engagement we would like to see as part of your GSOC. You already drew Emre’s attention, often discussions done in isolation undermine this impact. So good job on a start and we will expect over the summer you will engage lot more.
Application dependency libraries, compiler versions can quickly get tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod <https://github.com/TACC/Lmod>) within a VM to alleviate this but that will be an over kill. Eventually containers will hopefully alleviate this problem. > The worker VMs need to have the Mesos libraries installed and the Mesos > master needs to be configured with the IPs of the worker VMs. So, if new VMs > are created each time, we will develop the playbook for the creation of the > Mesos cluster on the fly, and then launch the application on it. Can the slave not register itself dynamically and report its IP Address? With either multiple applications per VM or one per VM, you will need to look into the capability of a persistent master and short lived elastic slave instances right? May be I am mis-understating your questions. Can you help clarify this more? Suresh > On Apr 27, 2016, at 1:21 PM, Emre Brookes <[email protected]> wrote: > > Pierce, Marlon wrote: >> Thanks, Gourav, this is great. How will the design differ in the two cases? >> >> Since installing the applications is not a simple task and there aren’t good >> docker libraries (for example) for many of them, I would assume that we >> would have a collection of VMs that have the codes already installed. > FWIW, that model would work well with GenApp usage also. Although, I can > imagine a scenario where a part of job initialization script (running under > the already "code installed" VM) checks for updates to the code and installs > them, but that can be independent of Airavata/Aurora/Mesos's knowledge. > > -Emre > >> >> Marlon >> >> >> From: Gourav Rattihalli <[email protected] >> <mailto:[email protected]>> >> Reply-To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Date: Wednesday, April 27, 2016 at 1:01 PM >> To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream >> >> Hi Dev's, >> >> I have been working on the integration of Apache Aurora and Mesos with >> Airavata and Jetstream. Using Mangirish's latest code, VMs can now be >> created using on Jetstream. I want to understand how Airavata is likely to >> use the VMs for applications. Will airavata re-use existing VMs for >> successive applications for a given community, or will new VMs be created >> for each application? This will decide how we design the automated creation >> of a Mesos cluster using the new VMs that are created. I understand that we >> will have a Mesos master VM that will run all the time on Jetstream. >> >> Here is the Job submission flow that I am assuming: >> >> Airavata Client--> Airavata >> Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will >> create .aurora configuration file that will be used to launch the >> job)-->Aurora/Mesos-->VMs >> >> So, I will design a module that will work with Orchestrator-->GFAC to >> generate the appropriate Apache Aurora script to be submitted to the Aurora >> master. >> >> Please let me know if there are any comments, suggestions on this plan. >> >> -- >> Regards, >> Gourav Rattihalli >
