On 04/11/2017 02:08 PM, Pierre Riteau wrote:
On 4 Apr 2017, at 22:23, Jay Pipes <[email protected]
<mailto:[email protected]>> wrote:
On 04/04/2017 02:48 PM, Tim Bell wrote:
Some combination of spot/OPIE
What is OPIE?
Maybe I missed a message: I didn’t see any reply to Jay’s question about
OPIE.
Thanks!
OPIE is the OpenStack Preemptible Instances
Extension: https://github.com/indigo-dc/opie
I am sure other on this list can provide more information.
Got it.
I think running OPIE instances inside Blazar reservations would be
doable without many changes to the implementation.
We’ve talked about this idea several times, this forum session would be
an ideal place to draw up an implementation plan.
I just looked through the OPIE source code. One thing I'm wondering is
why the code for killing off pre-emptible instances is being done in the
filter_scheduler module?
Why not have a separate service that merely responds to the raising of a
NoValidHost exception being raised from the scheduler with a call to go
and terminate one or more instances that would have allowed the original
request to land on a host?
Right here is where OPIE goes and terminates pre-emptible instances:
https://github.com/indigo-dc/opie/blob/master/opie/scheduler/filter_scheduler.py#L92-L100
However, that code should actually be run when line 90 raises NoValidHost:
https://github.com/indigo-dc/opie/blob/master/opie/scheduler/filter_scheduler.py#L90
There would be no need at all for "detecting overcommit" here:
https://github.com/indigo-dc/opie/blob/master/opie/scheduler/filter_scheduler.py#L96
Simply detect a NoValidHost being returned to the conductor from the
scheduler, examine if there are pre-emptible instances currently running
that could be terminated and terminate them, and re-run the original
call to select_destinations() (the scheduler call) just like a Retry
operation normally does.
There's be no need whatsoever to involve any changes to the scheduler at
all.
and Blazar would seem doable as long as the resource provider
reserves capacity appropriately (i.e. spot resources>>blazar
committed along with no non-spot requests for the same aggregate).
Is this feasible?
No. :)
As mentioned in previous emails and on the etherpad here:
https://etherpad.openstack.org/p/new-instance-reservation
I am firmly against having the resource tracker or the placement API
represent inventory or allocations with a temporal aspect to them (i.e.
allocations in the future).
A separate system (hopefully Blazar) is needed to manage the time-based
associations to inventories of resources over a period in the future.
Best,
-jay
I'm not sure how the above is different from the constraints I mention
below about having separate sets of resource providers for preemptible
instances than for non-preemptible instances?
Best,
-jay
Tim
On 04.04.17, 19:21, "Jay Pipes" <[email protected]
<mailto:[email protected]>> wrote:
On 04/03/2017 06:07 PM, Blair Bethwaite wrote:
> Hi Jay,
>
> On 4 April 2017 at 00:20, Jay Pipes <[email protected]
<mailto:[email protected]>> wrote:
>> However, implementing the above in any useful fashion requires
that Blazar
>> be placed *above* Nova and essentially that the cloud operator
turns off
>> access to Nova's POST /servers API call for regular users.
Because if not,
>> the information that Blazar acts upon can be simply
circumvented by any user
>> at any time.
>
> That's something of an oversimplification. A reservation system
> outside of Nova could manipulate Nova host-aggregates to "cordon
off"
> infrastructure from on-demand access (I believe Blazar already uses
> this approach), and it's not much of a jump to imagine operators
being
> able to twiddle the available reserved capacity in a finite cloud so
> that reserved capacity can be offered to the subset of
users/projects
> that need (or perhaps have paid for) it.
Sure, I'm following you up until here.
> Such a reservation system would even be able to backfill capacity
> between reservations. At the end of the reservation the system
> cleans-up any remaining instances and preps for the next
> reservation.
By "backfill capacity between reservations", do you mean consume
resources on the compute hosts that are "reserved" by this paying
customer at some date in the future? i.e. Spot instances that can be
killed off as necessary by the reservation system to free resources to
meet its reservation schedule?
> The are a couple of problems with putting this outside of Nova
though.
> The main issue is that pre-emptible/spot type instances can't be
> accommodated within the on-demand cloud capacity.
Correct. The reservation system needs complete control over a
subset of
resource providers to be used for these spot instances. It would
be like
a hotel reservation system being used for a motel where cars could
simply pull up to a room with a vacant sign outside the door. The
reservation system would never be able to work on accurate data unless
some part of the motel's rooms were carved out for reservation
system to
use and cars to not pull up and take.
> You could have the
> reservation system implementing this feature, but that would
then put
> other scheduling constraints on the cloud in order to be effective
> (e.g., there would need to be automation changing the size of the
> on-demand capacity so that the maximum pre-emptible capacity was
> always available). The other issue (admittedly minor, but still a
> consideration) is that it's another service - personally I'd love to
> see Nova support these advanced use-cases directly.
Welcome to the world of microservices. :)
-jay
_______________________________________________
OpenStack-operators mailing list
[email protected]
<mailto:[email protected]>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________
OpenStack-operators mailing list
[email protected]
<mailto:[email protected]>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
