Re: Questions about Aurora scheduling policy

[Riccardo Poggi]

Thanks Bill,

I will answer your questions directly, but it may
also make sense to have a higher-level discussion about your requirements
to possibly offer alternative approaches.
Sure, that sounds very interesting.
The main scheduling constraint in our system is that it cannot tolerate 
an undefined pending period, it needs at least some fast-feedback.

Example scenario: the system asks to launch N instances of a given 
process with a specified set of resource constraints, now the scheduler 
should either
- start them all successfully
- start only a part of them
    + because of resource saturation for example
- fail to start them
    + because those resources are not at all available in the farm, or 
they are effectively busy, or ...
without queuing, but just returning the operation result.


There are also other, more functional, requirements. The most 
interesting ones probably are:
* Hooks. For good integration with the infrastructure, like publishing 
of information, access management control, and so on...
* Fault tolerance and Error recovery. A failure in either the scheduler 
or executor shall not take down the managed processes and shall recover 
it after a restart for example.
* Ownership. Possibility to set uid owner of the underlying processes.
* Notification system. Provide informations about the jobs/processes 
status (both in pull or push mode)


3. Would it possible to have Aurora handle tasks with no resource
constraints?
[...] but the real question is:
what behavior do you want from scheduling without accounting?
Yes, that is indeed the question. What would be the default scheduling 
behaviour in case of resource abundance?
Would it be possible to impose a "spreading" constraint on a subset of 
the farm that a-priori I know is able to handle a defined set of jobs?


Cheers,
   Riccardo


On 11/24/2015 06:38 AM, Bill Farner wrote:
Welcome, happy to help!  I will answer your questions directly, but it may
also make sense to have a higher-level discussion about your requirements
to possibly offer alternative approaches.

1. Would it possible to subscribe to state change for a given job/task and
receive notifications?

Not today, but i'm very open to the idea.  I think there are some cool
things you could implement with this behavior.  A concern i often have,
though, is that consumers of this data cannot handle cases where an event
fails to be delivered (i.e. they want a replica of the scheduler's state).
At any rate, i'd love to offer this behavior!

2. Would it possible to set a Pending time-out for tasks that take too long
to be Assigned?

Not currently.  You could implement this by polling the API and killing
tasks that took too long to schedule.  This would allow you to decide how
to react (if at all).

3. Would it possible to have Aurora handle tasks with no resource
constraints?

No.  Both Aurora and Mesos require CPU and memory to be specified for
tasks.  A client of Aurora could choose defaults, but the real question is:
what behavior do you want from scheduling without accounting?


On Mon, Nov 23, 2015 at 5:34 PM, Riccardo Poggi <riccardo.po...@cern.ch>
wrote:

Hello,

In order to introduce Aurora into our distributed system we would like to
have it slowly, and hopefully transparently, replace what is currently the
process manger component. To do that it would have to programmatically
interface with other parts of the system that are, at the moment, taking
care of what it could be considered the "active" orchestration.

I've tried Aurora and looked at the docs, but I'm still left with some
open questions:

1. Would it possible to subscribe to state change for a given job/task and
receive notifications?

2. Would it possible to set a Pending time-out for tasks that take too
long to be Assigned?

3. Would it possible to have Aurora handle tasks with no resource
constraints?


Thanks,
   Riccardo