Hi Suresh, First of all I would like to commend you for starting this discussion. This > is exactly kind of the mailing list engagement we would like to see as part > of your GSOC. You already drew Emre’s attention, often discussions done in > isolation undermine this impact. So good job on a start and we will expect > over the summer you will engage lot more. > > Thanks. Yes, I plan to actively use this mailing list to keep my design focused on real use cases, and also to get regular feedback.
> Application dependency libraries, compiler versions can quickly get > tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod) > within a VM to alleviate this but that will be an over kill. Eventually > containers will hopefully alleviate this problem. > > The worker VMs need to have the Mesos libraries installed and the Mesos > master needs to be configured with the IPs of the worker VMs. So, if new > VMs are created each time, we will develop the playbook for the creation of > the Mesos cluster on the fly, and then launch the application on it. > > > Can the slave not register itself dynamically and report its IP Address? > With either multiple applications per VM or one per VM, you will need to > look into the capability of a persistent master and short lived elastic > slave instances right? May be I am mis-understating your questions. Can you > help clarify this more? > > Yes, this is possible, and was the first of the two options I had emailed. To support this kind of elasticity, we will work on developing a script/playbook to install the Mesos libraries on the new slaves/agents and connect them with the master. Thanks, Gourav On Apr 27, 2016, at 1:21 PM, Emre Brookes <[email protected]> wrote: Pierce, Marlon wrote: Thanks, Gourav, this is great. How will the design differ in the two cases? Since installing the applications is not a simple task and there aren’t good docker libraries (for example) for many of them, I would assume that we would have a collection of VMs that have the codes already installed. FWIW, that model would work well with GenApp usage also. Although, I can imagine a scenario where a part of job initialization script (running under the already "code installed" VM) checks for updates to the code and installs them, but that can be independent of Airavata/Aurora/Mesos's knowledge. -Emre Marlon From: Gourav Rattihalli <[email protected] < mailto:[email protected] <[email protected]>>> Reply-To: "[email protected] <mailto:[email protected] <[email protected]>>" <[email protected] < mailto:[email protected] <[email protected]>>> Date: Wednesday, April 27, 2016 at 1:01 PM To: "[email protected] <mailto:[email protected] <[email protected]>>" <[email protected] < mailto:[email protected] <[email protected]>>> Subject: Design of Mesos/Aurora integration with Airavata on Jetstream Hi Dev's, I have been working on the integration of Apache Aurora and Mesos with Airavata and Jetstream. Using Mangirish's latest code, VMs can now be created using on Jetstream. I want to understand how Airavata is likely to use the VMs for applications. Will airavata re-use existing VMs for successive applications for a given community, or will new VMs be created for each application? This will decide how we design the automated creation of a Mesos cluster using the new VMs that are created. I understand that we will have a Mesos master VM that will run all the time on Jetstream. Here is the Job submission flow that I am assuming: Airavata Client--> Airavata Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will create .aurora configuration file that will be used to launch the job)-->Aurora/Mesos-->VMs So, I will design a module that will work with Orchestrator-->GFAC to generate the appropriate Apache Aurora script to be submitted to the Aurora master. Please let me know if there are any comments, suggestions on this plan. -- Regards, Gourav Rattihalli -- Regards, Gourav Rattihalli
