Re: Aurora now supports multiple executors

2016-08-15 Thread meghdoot_b
Renan shared it in user list. Here you go https://github.com/rdelval/gorealis/blob/master/docs/getting-started.md More features are getting added. Thx PS: btw compose executor docs would be updated as well (it's stale) and it's going through production hardening. Sent from my iPhone > On Au

Re: Aurora now supports multiple executors

2016-08-05 Thread meghdoot_b
Expect that to be up by mid next week and put in 0.16 docs. Sent from my iPhone > On Aug 5, 2016, at 9:23 AM, Zhitao Li wrote: > > This sounds awesome! > > Is there a document for how to use the feature? > > On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya < > meghdoo...@yahoo.com.inval

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

2016-06-13 Thread meghdoot_b
Got it. Thx Max. Renan will starting working on 2 and 3, involving and updating you all. Thx Sent from my iPhone > On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko wrote: > > David pinged me offline suggesting I misread the proposal for (2) as > an attempt to support multiple executors per task

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

2016-06-13 Thread meghdoot_b
I will leave Renan for a detailed response. If aurora can schedule Job A with thermos and Job B that requires a different executor that is a first class concept in mesos, why is there a objection? Renan's patch of replacing thermos executor is already in aurora. However right now aurora can sta

Re: Aurora performance impact with hourly query runs

2016-06-12 Thread meghdoot_b
Yes Maxim really appreciate the tip. That's quiet a difference. One follow up question, any reason of not making MemTaskStore the default in aurora? Thx Sent from my iPhone > On Jun 12, 2016, at 9:48 AM, Shyam Patel wrote: > > The query performance improved drastically, It took only 29ms for

Re: Multi-role cluster doc

2016-01-17 Thread meghdoot_b
This would indeed be useful. Thx > On Jan 15, 2016, at 9:27 AM, Bill Farner wrote: > > Hey everyone, > > Now that we've landed the patch below (thanks again, Zhitao!), it dawned on > me that we should have a bit of documentation providing guidance for using > Aurora in a cluster with other fra

Re: Checkpoint

2015-11-19 Thread meghdoot_b
I started digging the code and found the same as well. Thx for confirming Bill. General question, does the framework failover timeout feature only work if checkpoint is set by framework as well? Or the checkpoint feature is strictly for slave side and mesos master will keep tasks running regard

Checkpoint

2015-11-19 Thread meghdoot_b
I am guessing aurora does not use mesos checkpoint feature where tasks can run even if slave stopped (for an upgrade say). Can this be supported (optionally) if not today especially as part of custom executor support? Mesos slaves has enabled check pointing by default a while back but it needs f

Re: Multiple executor support

2015-11-08 Thread meghdoot_b
Read "all" as "most" for available execution parameters for the executor to act on. Few like the user can be controlled from operator like you mention. Thx Sent from my iPhone On Nov 8, 2015, at 10:50 AM, Bill Farner wrote: >> all standard mesos inputs are available to the custom executor

Re: Multiple executor support

2015-11-08 Thread meghdoot_b
I see. That would be excellent Bill if you are willing to take on this and actively drive it. As you mention if all standard mesos inputs are available to the custom executor which is driven from client that would be great. URI set for files in sandbox, data blob (could be anything that an exe

Re: Multiple executor support

2015-11-04 Thread meghdoot_b
Could you kindly elaborate with an example how the end user can pass in the executor of their choice in the API . I remember that other than a name passed on the API there was no scope to pass in the path of the custom executor or things like Fetcher URI sets that can be downloaded by mesos in t

Re: Multiple executor support

2015-11-03 Thread meghdoot_b
Bill what do you think? Thx Sent from my iPhone > On Nov 2, 2015, at 1:00 PM, Zameer Manji wrote: > > +wfarner > > I believe Bill was heavily involved in reviewing the proposed patch and > design. Bill, care to comment on what you think here? > >> On Mon, Nov 2, 2015 at 12:55 PM, wrote: >>

Multiple executor support

2015-11-02 Thread meghdoot_b
Do we have a decision on this? https://issues.apache.org/jira/plugins/servlet/mobile#issue/AURORA-1376 It would help to know where we stand on this. Thx Sent from my iPhone

Re: Unified container support in Aurora

2015-10-21 Thread meghdoot_b
Is the file system isolation going to provide aufs/overlayfs etc kind of drivers to store the layers to Steve's point with COW because that is a key part of docker performance? Also docker has started to support cgroup parents flag (has some issues) but once fixed can be used to form a cgroups

Re: JobConfig diff API

2015-09-15 Thread meghdoot_b
Isn't this data supposed to be any blob that transparently passes in to the executor through mesos data blob. Why would we want to impose any sort of format? It could be a binary blob. Executor writers should be able to move between different schedulers/frameworks without any rework ideally. Thi

Re: Using a config file to support custom executors: potential paradigm shift

2015-07-08 Thread meghdoot_b
In general the number of executors supported would be not very dynamic but as needed in that deployment environment. The config file listing all the supported executors can be simply put in the deb or rpm package of the scheduler for example. If values are fairly dynamic then the mesos approach