Re: Support instance-specific TaskConfig in CreateJob API

2016-08-15 Thread meghdoot bhattacharya
e. One can use the instanceId in this case as well to grab configs, edit the final compose file for example in your use case. Thx From: Mauricio Garavaglia To: [email protected] Sent: Monday, August 15, 2016 11:12 AM Subject: Re: Support instance-specific TaskConfig in CreateJob

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-15 Thread Mauricio Garavaglia
On Mon, Aug 15, 2016 at 2:49 PM, David McLaughlin wrote: > On Mon, Aug 15, 2016 at 10:40 AM, Mauricio Garavaglia < > [email protected]> wrote: > > > Hi, > > > > WRT the constraint use case: > > > > Let's say I have two instances of a service, these instances would need > > different Do

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-15 Thread Joshua Cohen
On Fri, Aug 12, 2016 at 11:05 AM, Min Cai wrote: > > > (1) There will be a spam of jobs with each only have one task instance. > This will make the Aurora UI almost unusable. Also, the user will have to > track the grouping of jobs and introduce a "job group" like concept on the > client side. I

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-15 Thread David McLaughlin
On Mon, Aug 15, 2016 at 10:40 AM, Mauricio Garavaglia < [email protected]> wrote: > Hi, > > WRT the constraint use case: > > Let's say I have two instances of a service, these instances would need > different Docker arguments: > But that's less about using Docker and more how you're us

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-15 Thread Mauricio Garavaglia
Hi, WRT the constraint use case: Let's say I have two instances of a service, these instances would need different Docker arguments: - Labels. Log drivers uses container labels to identify who is producing this logs. The container name doesn't work with Mesos of course. So you have loglabel=serv

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-15 Thread Maxim Khutornenko
I would love to hear more about constraint use cases that don't work across jobs to see if/how we can extend Aurora to support them. As far as heterogeneous jobs go, that effort would require rethinking quite a few assumptions around fundamental Aurora principles to ensure we don't lock ourselves

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-13 Thread Mauricio Garavaglia
Hi, We have been experimenting with the idea of having heterogeneous tasks in a job. Mainly to support different docker container configurations (like volumes to let tasks have different storage, different labels for logging purposes, or ip addresses). The main reason for using this instead of sep

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-12 Thread Min Cai
Thanks Maxim. Please see my previous email to David's comments for more detailed response. On Fri, Aug 12, 2016 at 9:24 AM, Maxim Khutornenko wrote: > I am cautious about merging createJob and startJobUpdate as we don't > support updates of adhoc jobs. It's logically unclear what adhoc job updat

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-12 Thread Min Cai
Hi David, Thanks for your quick comments. Please see my response inline. On Fri, Aug 12, 2016 at 8:03 AM, David McLaughlin wrote: > Hi Min, > > I'd prefer to add support for ad-hoc jobs to startJobUpdate and completely > remove the notion of job create. > > There are a few reasons that we can n

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-12 Thread Maxim Khutornenko
I am cautious about merging createJob and startJobUpdate as we don't support updates of adhoc jobs. It's logically unclear what adhoc job update would mean as adhoc job instances are not intended to survive terminal state. Even if we decided to do so I am afraid it would not help with the scenario

Re: Support instance-specific TaskConfig in CreateJob API

2016-08-12 Thread David McLaughlin
Hi Min, I'd prefer to add support for ad-hoc jobs to startJobUpdate and completely remove the notion of job create. " Also, even the > StartJobUpdate API is not scalable to a job with 10K ~ 100K task instances > and each instance has different task config since we will have to invoke > StartJobUp