Hi Oliver, GoDocker looks very promising. I have been looking for a a framework to substitute AWS Lambda for batch jobs scheduling based on docker for flexibility. That said, I would like to know your thought on this to better understand if GoDocker could do the job. A couple of things I would need to know better:
- How fast can GoDocker launch a job? Can GoDocker pre populate all nodes with the container and files needed for the job to achieve <500ms launches? I realize this depends a lot on the Job size, just looking for a generic view before getting into full testing. - Is it possible to extend GoDocker to enable usage metrics reports? This could be achieved by simply logging a few metrics relating to the user and job. { name: “João Ribeiro”, email: “jonnyb...@gmail.com <mailto:jonnyb...@gmail.com>”, phone: “+351 916 002 675 <tel:+351916002675>”, skype: “jmn_ribeiro <skype:jmn_ribeiro?add>”, linkedin: “https://pt.linkedin.com/in/jmnribeiro <https://pt.linkedin.com/in/jmnribeiro>” github: “https://github.com/JonnyBGod <https://github.com/JonnyBGod>” } > On 09 Dec 2015, at 10:22, Elouan Keryell-Even <elouan.kery...@gmail.com> > wrote: > > Thanks for the insight, very interesting indeed. > > Elouan Keryell-Even > Software Engineer @ Atos Integration > > Toulouse, France > > > 2015-12-09 11:08 GMT+01:00 Olivier Sallou <olivier.sal...@irisa.fr > <mailto:olivier.sal...@irisa.fr>>: > > > De: "Elouan Keryell-Even" <elouan.kery...@gmail.com > <mailto:elouan.kery...@gmail.com>> > À: user@mesos.apache.org <mailto:user@mesos.apache.org> > Envoyé: Mercredi 9 Décembre 2015 10:48:10 > Objet: Re: GenOuest makes use of Mesos > > Hi Olivier, > > I've just read the presentation on your project's webpage, and it seems cool! > I'm curious about the following mentioned feature: "Optional mount of user > home or other shared directories in container". Does your framework take care > of remotely copying the home directory onto the node where the container is > going to run, or does the directory have to be already available (as a shared > directory for example). > We focus to "replace" classical computing cluster frameworks like Sun Grid > Engine. So home directories etc... are shared among nodes (nfs, etc...), like > in a usual cluster environment. > The mount of "user home dir" is based on an Auth/ACL plugin/configuration. > Admin specifies the available mounts for all users, or on a project basis. > Then the Auth/ACL plugin maps those volumes to real available directories (if > using the LDAP auth plugin, then get homeDirectory from LDAP and add as a > volume to container). > Otherwise, we are currently using framework Chronos for our Batch oriented > containers, and your framework seems to fit the same spot. If you have some > experience with Chronos, I'd be interested in a brief comparison of both > frameworks. From what I understand, GODocker offers scheduling policy's > customization, which I don't think Chronos does. > There is indeed scheduling algorithms plugin mechanism. An interesting one is > the fair-share policy imlpementation, where tasks are scheduler (ordered) > based on previous user/project consumption. > > Difference with Chronos, is Chronos is made for cron like jobs. GODocker is a > batch submission tool. User specifies the computing script he wants to > execute and it is executed "immediatly". The scheduling is a matter of > putting priorities when executing jobs (if users submits 1000 jobs but I have > only 100 slots available). We focus on users submitting "many" jobs or many > users submitting jobs, each job could be a computing task of a few seconds or > several days. > > > > Olivier > > > Thanks for your attention, > > Elouan Keryell-Even > Software Engineer @ Atos Integration > > Toulouse, France > > > 2015-12-08 14:49 GMT+01:00 Olivier Sallou <olivier.sal...@irisa.fr > <mailto:olivier.sal...@irisa.fr>>: > Hi, > the GenOuest (http://www.genouest.org <http://www.genouest.org/>) academic > lab is now using Mesos > in production in its core facility to manage scientists computing tasks > (for bioinformatics) > To do so, we have developed a new mesos framework, GoDocker > (http://www.genouest.org/godocker <http://www.genouest.org/godocker>) to > submit batch computing scripts on > premises. It mounts users home directory or other shared resources to > execute jobs in Docker containers, using Mesos as main scheduler. > GoDocker schedules the jobs according to user/groups priorities and > quotas and provides a CLI, a REST web interface and a partial DRMAA > library support. Framewok is open source. > > Thanks for adding us to the Mesos fellows ;-) > > Regards > > Olivier (GenOuest core developer member) > > -- > Olivier Sallou > IRISA / University of Rennes 1 > Campus de Beaulieu, 35000 RENNES - FRANCE > Tel: 02.99.84.71.95 <tel:02.99.84.71.95> > > gpg key id: 4096R/326D8438 (keyring.debian.org <http://keyring.debian.org/>) > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > > > >
signature.asc
Description: Message signed with OpenPGP using GPGMail