Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Jay Pipes]

On 03/16/2016 07:10 AM, Attila Fazekas wrote:

NO : For any kind of extra quota service.

In other places I saw other reasons for a quota service or similar,
  the actual cost of this approach is higher than most people would think so NO.

Maybe Library,
But I do not want to see for example the bad pattern used in nova to spread 
everywhere.

The quota usage handling MUST happen in the same DB transaction as the
resource record (volume, server..) create/update/delete  .


No, it doesn't.

This isn't even a remotely realistic thing for a distributed system like 
Nova to try. There often isn't a single DB transaction that creates or 
updates or deletes something in Nova. API calls typically traverse 3 or 
more separate compute and controller node services, saving state 
transitions to a resource record sometimes *dozens* of times over the 
course of the API call -- many of which are entirely asynchronous.


In addition to the already-distributed, mostly-asynchronous nature of 
Nova, keep in mind that a Nova deployment can have multiple cells each 
having its own entirely different database, multiple availability zones 
each with multiple cells, and multiple regions each with multiple 
availability zones.


You still want to keep a database transaction (two phase commit or 
otherwise) open for the duration of events that cross any of those 
boundaries?


No, I think not.


There is no need for.:
- reservation-expirer services or periodic tasks ..
- there is no need for quota usage correcter shell scripts or whatever
- multiple commits

We have a transaction capable DB, to help us,
not using it would be lame.


We *do* use transactions in the quota system. That's not the primary 
problem with the Nova quota system.


The major problem with the Nova quota system is that it keeps a cache of 
usage records in its quota_usages (and reservations) table instead of 
querying the primary tables that *actually* store real resources.


Any time you introduce cache tables, you introduce greater potential for 
races to occur.


Either you accept the inevitability of those race conditions, or you 
relax the constraints that you put on the locking system, and/or you get 
rid of the caching and accept some small degradation of raw performance 
of quota checks.


In my opinion, we should have a system that issues a single quota check 
against the actual resource records. [1] This check should occur right 
up front before any operation that would consume a resource. [2]


I don't actually think that this proposed quota thing should be service. 
Why? Because a separate service would inevitably be implemented by 
creating cache tables about quota usages. And that's at the root of the 
problem with race conditions in existing quota solutions in OpenStack.


Better to create a library that actually has no database of its own at 
all. Instead, have the library expose a simple and consistent API that 
relies on an implementation plugin that would live in each OpenStack 
component that needed quota checks. These plugins would query the 
*actual* resource tables that exist in their own databases to determine 
if a user's request can be satisfied.


We'd still need some central-ish place to *update* quota amounts for 
individual users/tenants/defaults. IMHO, Keystone is the logical place 
to put this kind of information. No need for a new service when we 
already have one that can serve this purpose. A previous suggestion of 
just using the auth token to either store the quotas themselves or a 
link to grab quota information for a user is a good suggestion.


Finally, someone had mentioned rate limiting in an earlier response. I 
don't believe OpenStack should be in the business of rate limiting. 
There are tested and scalable solutions that do rate limiting for HTTP 
requests. [3] Those should be used instead of Python code in a WSGI 
application or middleware service.


My two cents,
-jay

[1] In Nova, this would be the new allocations table in the 
resource-providers modeling or the instances table in the legacy modeling


[2] If there's no cache table of quota usages, there's NO reason why 
quota calculations need to be made on ANY request that doesn't actually 
consume a new resource. This means no quota calculations on DELETE 
operations, no quota calculations on most UPDATE operations (since the 
amount of consumed resources generally doesn't change at all)


[3]
https://lincolnloop.com/blog/rate-limiting-nginx/
http://www.openrepose.org/
https://tyk.io/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Nikhil Komawar]

Thanks for your feedback Erno. I've answered your question inline.

On 3/17/16 2:21 AM, Erno Kuvaja wrote:
> On Wed, Mar 16, 2016 at 6:25 AM, Nikhil Komawar  > wrote:
>
> Hello everyone,
>
>
>
> tl;dr;
>
> I'm writing to request some feedback on whether the cross project
> Quotas
>
> work should move ahead as a service or a library or going to a far
>
> extent I'd ask should this even be in a common repository, would
>
> projects prefer to implement everything from scratch in-tree?
> Should we
>
> limit it to a guideline spec?
>
>
>
> But before I ask anymore, I want to specifically thank Doug Hellmann,
>
> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
>
> Laski for the early feedback that has helped provide some good
> shape to
>
> the already discussions.
>
>
>
> Some more context on what the happenings:
>
> We've this in progress spec [1] up for providing context and platform
>
> for such discussions. I will rephrase it to say that we plan to
>
> introduce a new 'entity' in the Openstack realm that may be a
> library or
>
> a service. Both concepts have trade-offs and the WG wanted to get more
>
> ideas around such trade-offs from the larger community.
>
>
> Would you mind to expand this "we" here?
>  

Quota WG (
http://eavesdrop.openstack.org/#Cross-project_Quotas_and_Nested_Quotas_Working_Group_Virtual_Standup
)

>
> Service:
>
> This would entail creating a new project and will introduce managing
>
> tables for quotas for all the projects that will use this service. For
>
> example if Nova, Glance, and Cinder decide to use it, this
> 'entity' will
>
> be responsible for handling the enforcement, management and DB
> upgrades
>
> of the quotas logic for all resources for all three projects. This
> means
>
> less pain for projects during the implementation and maintenance
> phase,
>
> holistic view of the cloud and almost a guarantee of best practices
>
> followed (no clutter or guessing around what different projects are
>
> doing). However, it results into a big dependency; all projects
> rely on
>
> this one service for right enforcement, avoiding races (if do not
>
> incline on implementing some of that in-tree) and DB
>
> migrations/upgrades. It will be at the core of the cloud and prone to
>
> attack vectors, bugs and margin of error.
>
>
> I'd prefer not. As lots of concern raised already ref. latency, extra
> api etc.
> Based on the unifying the user interface, common api might be desired
> option, but it's own service, not so much.
>  
>
>
> Library:
>
> A library could be thought of in two different ways:
>
> 1) Something that does not deal with backed DB models, provides a
>
> generic enforcement and management engine. To think ahead a little bit
>
> it may be a ABC or even a few standard implementation vectors that can
>
> be imported into a project space. The project will have it's own
> API for
>
> quotas and the drivers will enforce different types of logic; per se
>
> flat quota driver or hierarchical quota driver with custom/project
>
> specific logic in project tree. Project maintains it's own DB and
>
> upgrades thereof.
>
>
> Partially decent idea, just the fact annoys me that this is climbing
> the tree
> arse ahead. The individual API's is perhaps the worst option with common
> code for the quotas as each project has their own requirements where the
> API might be generalized.
>
>
> 2) A library that has models for DB tables that the project can import
>
> from. Thus the individual projects will have a handy outline of
> what the
>
> tables should look like, implicitly considering the right table
> values,
>
> arguments, etc. Project has it's own API and implements drivers
> in-tree
>
> by importing this semi-defined structure. Project maintains it's own
>
> upgrades but will be somewhat influenced by the common repo.
>
>
> This is really not benefitting anyone. Again each project has their own
> requirements for quotas, while the user experience is the one thing we
> should try to unify. I have really difficulties to see Zaqar, Nova and
> Glance
> fitting into the single quota model, while the API interacting with
> those could
> be similar.
>
>
> Library would keep things simple for the common repository and
> sourcing
>
> of code can be done asynchronously as per project plans and priorities
>
> without having a strong dependency. On the other hand, there is a
>
> likelihood of re-implementing similar patterns in different projects
>
> with individual projects taking responsibility to keep things up to
>
> date. Attack vectors, bugs and margin of error are project
> responsibilities
>
>
> This is the problem I see with oslo 

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Nikhil Komawar]

Hi Amrith,

Thank you for your feedback and asking all the important questions. Over
the last week, I've interacted with people who are willing to work on
this, who have been involved in this, who have stake in this, etc. Based
on the digest of all of those I intend to reply to your questions.
Again, this initiative is still evolving and gladly we've a cross
project session proposed for the Austin summit [1].

Please find my responses inline.

[1] https://etherpad.openstack.org/p/newton-cross-project-sessions

On 3/16/16 8:27 AM, Amrith Kumar wrote:
> Nikhil, thank you for the very timely posting. This is a topic that has
> been discussed quite a bit recently within the Trove team. I've read the
> document you reference as [1] and I have not been part of earlier
> conversations on this subject so I may be missing some context here.
>
> I feel that the conversation (in [1], in the email thread) has gone to a
> discussion of implementation details (library vs. service, quota
> enforcement engine, interface, ...) when I think there is still some
> ambiguity in my mind about the requirements. What is it that this
> capability will provide and what is the contract implied when a service
> adopts this model.
>

I had raised two questions 1) if we needed the specifics as a part of
the cross project spec (in general) and 2) how do these implementations
get there. I think the intent is to build on those details and not
necessarily provide a authority over them. However, the biggest voices
are saying it's a hard code to get right. So, the spec will need to
updated to reflect accordingly -- whichever has been the successful
attempt of getting it right.

> For example, consider this case that we see in Trove. In response to a
> user request to create a cluster of databases, Trove must provision
> storage (cinder), compute (nova), networks (let's say neutron), and so
> on. As stated by Boris in his email, it would be ideal if Trove had a
> confirmation from all projects that there was quota available for the
> requests that would be made before the requests actually are made. This
> implies therefore that participating projects (cinder, nova, neutron,
> ...) would have to support some reservations scheme and subsequently
> honor requests based on a reservation. So, I think there's more to this
> than just another library or project, there's an implication for
> projects that wish to participate in this scheme. Or am I wrong in this
> understanding?

I think the eventual plan is to get to this state. But building
everything from scratch will mean getting things right first and then
incrementally improving!

> Several have observed that we have a 'transaction capable DB to help
> us'. Let me go further and say that we have a distributed transaction
> capable DB (yes, if you squint just the right way, MySQL can do 2phase
> commit) and we should leverage that to the hilt.
>
> But, I'd like to get caught up on the requirements that this capability
> is to provide before we get into implementations, so if there are any
> descriptions of those requirements, I'd appreciate getting pointers to that.
>

Yes, the PWG is helping build a user story around the requirements. But
those will be more on the conceptual level and probably what you are
looking for to ensure that it accommodates yours?

I would love for you to be involved in the discussions and it would be
nice if you can join the Quota WG standup (
http://eavesdrop.openstack.org/#Cross-project_Quotas_and_Nested_Quotas_Working_Group_Virtual_Standup
) once in a while on Mondays to bring in your perspective.

> Thanks,
>
> -amrith
>
>
> [1] https://review.openstack.org/#/c/284454/
>
> --
> Amrith Kumar, CTO   | amr...@tesora.com
> Tesora, Inc | @amrithkumar
> 125 CambridgePark Drive, Suite 400  | http://www.tesora.com
> Cambridge, MA. 02140| GPG: 0x5e48849a9d21a29b 
>
> On 03/16/2016 02:25 AM, Nikhil Komawar wrote:
>> Hello everyone,
>>
>> tl;dr;
>> I'm writing to request some feedback on whether the cross project Quotas
>> work should move ahead as a service or a library or going to a far
>> extent I'd ask should this even be in a common repository, would
>> projects prefer to implement everything from scratch in-tree? Should we
>> limit it to a guideline spec?
>>
>> But before I ask anymore, I want to specifically thank Doug Hellmann,
>> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
>> Laski for the early feedback that has helped provide some good shape to
>> the already discussions.
>>
>> Some more context on what the happenings:
>> We've this in progress spec [1] up for providing context and platform
>> for such discussions. I will rephrase it to say that we plan to
>> introduce a new 'entity' in the Openstack realm that may be a library or
>> a service. Both concepts have trade-offs and the WG wanted to get more
>> ideas around such trade-offs from the larger community.
>>
>> Service:
>> 

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Fox, Kevin M]

As an Op, I've seen quota's in the db get out of sync all the time I'm sure 
some of my clouds right now have some stuff out of sync :/

With my dev hat on:
The code is really hard to get right.

There are also a lot of places quota's are missing. quota's per host aggregate, 
quota's for floating ips per external network.

Also hierarchical quota's are going to be important and that will touch a 
million projects too.

Then there's lots of projects that don't implement but the most basic of 
quota's.

Really, centralizing it will be a lot of work initially, but long term, I think 
will pay off in that there will be only one place to fix bugs and add important 
features. It will allow each project to easily add quota's to the places they 
need to be in, since they won't have to write a whole quota system themselves.

With an op hat on again, putting them all in one place will also make it easier 
to administer. "floating ip's on this cloud are ... neutron, so I need to use 
the neutron command to set the quota for that" can be a thing of the past.

Thanks,
Kevin


From: Joshua Harlow [harlo...@fastmail.com]
Sent: Wednesday, March 16, 2016 9:58 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

On 03/16/2016 05:22 AM, Julien Danjou wrote:
> On Wed, Mar 16 2016, Attila Fazekas wrote:
>
>> The quota usage handling MUST happen in the same DB transaction as the
>> resource record (volume, server..) create/update/delete  .
>
> […]
>
>> We have a transaction capable DB, to help us,
>> not using it would be lame.
>
> Amen to that.

Double amen!

>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Chris Dent]

On Wed, 16 Mar 2016, Salvatore Orlando wrote:


- Looking at quotas it is worth distinguishing between management (eg::
resource limits per tenant and/or users), and enforcement (eg.: can the
bakery service give me 4 cookies or did I already eat too many?)
 While for the reasons listed throughout this thread the latter should
really happen in the same context where the request is going to served,
quota management instead might be its own service, or however being done in
a common endpoint for all OpenStack resources.


Thanks. I've been reading through this thread getting increasingly
confused. The proposed openstack-spec[1] does a pretty good job of
keeping management and enforcement separate but this thread, it's
not too clear. So somebody help me out, is the following in the
realm of right?

As stated by a few other people, enforcement is in the domain of
authZ: entity X has a "quota", or is a member of something that has
a "quota". Does their request + their existing use fit within that
quota? No, sorry, computer says no. That quota could be a fairly static
statement of a limit, however the information that needs to be expressed
there is probably complex.

The accounting of "their existing use" is the hard part, right? It is
also a generic problem we have in much of OpenStack:

* how much stuff is there?
* how much of it is used?
* who/what has used it?

Quota enforcement needs that information, schedulers/placers needs that
information. If that information were available as a service,
everybody would find it helpful.

If there are going to be services, there are two of them:

* accounting
* quota enforcement

Is that right?


- It has also been raised a good point about securing a chunk of resources
across project, that is also related to John's point about business
quotas... I'm not sure it is necessary, but Blazar [1] kind of achieves
this - even if it was conceived with different purposes.


I believe this is described as 'reservations' in the spec and I
agree with John Garbutt that it greatly raises the complexity. If we
have a system where we're having to do claims, instead of just
reflecting reality (i.e. real usage), things will go wrong, quickly.

We're much better figuring out a simpler system that includes the
possibility to fail fast.

[1] The proposed spec: https://review.openstack.org/#/c/284454/

--
Chris Dent   (╯°□°)╯︵┻━┻http://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Hayes, Graham]

On 16/03/2016 06:28, Nikhil Komawar wrote:
> Hello everyone,
>
> tl;dr;
> I'm writing to request some feedback on whether the cross project Quotas
> work should move ahead as a service or a library or going to a far
> extent I'd ask should this even be in a common repository, would
> projects prefer to implement everything from scratch in-tree? Should we
> limit it to a guideline spec?



> Service:
> This would entail creating a new project and will introduce managing
> tables for quotas for all the projects that will use this service. For
> example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> be responsible for handling the enforcement, management and DB upgrades
> of the quotas logic for all resources for all three projects. This means
> less pain for projects during the implementation and maintenance phase,
> holistic view of the cloud and almost a guarantee of best practices
> followed (no clutter or guessing around what different projects are
> doing). However, it results into a big dependency; all projects rely on
> this one service for right enforcement, avoiding races (if do not
> incline on implementing some of that in-tree) and DB
> migrations/upgrades. It will be at the core of the cloud and prone to
> attack vectors, bugs and margin of error.

In my view:

Pros:

1. Improved UX
   - Currently it is difficult to have a common method of updating
 and viewing project quotas is difficult, and (last time I looked)
 impossible in Horizon.
2. Much easier to iterate on quotas - e.g. Nested Quotas

Cons:

1. Latency
2. Yet another thing in the critical path of API requests
3. Keeping compatibility across releases to allow for phased upgrades
could be problematic
4. How does the big tent feed into this - is it plugin based (which
increases the complexity of deploying additional OpenStack services)
or does this service have all projects in tree?

I know that there is probably tons of problems with this idea, but
something occurred to me, could we have this as part of keystone?

When a user gets a token, as part of the token they get a quota
object. It does a few things:

1. Removes the need for another service
2. Allows services to use an already existing interface
3. Still has a central place for updates / querying

It also has a few downsides that I can think of off the top of my head:

1. Mainly - would Keystone even be willing to take this as part of
their mission statement?
2. PKI tokens would have to be re-issued to get new quotas
3. PKI Tokens would have to be invalidated when decreasing tokens
4. The token size might explode for PKI
5. Extra load on Keystone


> Library:
> A library could be thought of in two different ways:
> 1) Something that does not deal with backed DB models, provides a
> generic enforcement and management engine. To think ahead a little bit
> it may be a ABC or even a few standard implementation vectors that can
> be imported into a project space. The project will have it's own API for
> quotas and the drivers will enforce different types of logic; per se
> flat quota driver or hierarchical quota driver with custom/project
> specific logic in project tree. Project maintains it's own DB and
> upgrades thereof.
> 2) A library that has models for DB tables that the project can import
> from. Thus the individual projects will have a handy outline of what the
> tables should look like, implicitly considering the right table values,
> arguments, etc. Project has it's own API and implements drivers in-tree
> by importing this semi-defined structure. Project maintains it's own
> upgrades but will be somewhat influenced by the common repo.

This would actually work quite well for us in Designate as the quotas
component is a plugin, we could switch to a openstack-common style
system quite easily.

It does not help in the UX side of things - but seems like it is the
quickest route to something being completed.

> Library would keep things simple for the common repository and sourcing
> of code can be done asynchronously as per project plans and priorities
> without having a strong dependency. On the other hand, there is a
> likelihood of re-implementing similar patterns in different projects
> with individual projects taking responsibility to keep things up to
> date. Attack vectors, bugs and margin of error are project responsibilities
>
> Third option is to avoid all of this and simply give guidelines, best
> practices, right packages to each projects to implement quotas in-house.
> Somewhat undesirable at this point, I'd say. But we're all ears!
>
> Thank you for reading and I anticipate more feedback.
>
> [1] https://review.openstack.org/#/c/284454/
>

Thanks

- Graham


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Joshua Harlow]

This has been proposed a number of times in the past with projects such as Boson
(https://wiki.openstack.org/wiki/Boson) and an extended discussion at one of the
summits (I think it was San Diego).

Then, there were major reservations from the PTLs at the impacts in terms of
latency, ability to reconcile and loss of control (transactions are difficult, 
transactions
across services more so).


Understood and I get the worry that this causes people.

But just some food for thought, I have heard from a grapevine this is 
how a company that starts with 'g' and ends with 'oogle' does quota on 
there resources. A service (I don't know the name) manages the quotas 
and other services can subscribe to that other services events (to say 
sync themselves with that service).


But this is just the grapevine, so the information may be a little 
distorted and/or not correct, ha...


-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[gordon chung]

On 16/03/2016 9:39 AM, Sean Dague wrote:
> This has to live inside all the upgrade constraints we currently have
> (like online data migration in the Nova case), otherwise it's a non starter.

completely agree

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Erno Kuvaja]

On Wed, Mar 16, 2016 at 6:25 AM, Nikhil Komawar 
wrote:

> Hello everyone,
>
> tl;dr;
> I'm writing to request some feedback on whether the cross project Quotas
> work should move ahead as a service or a library or going to a far
> extent I'd ask should this even be in a common repository, would
> projects prefer to implement everything from scratch in-tree? Should we
> limit it to a guideline spec?
>
> But before I ask anymore, I want to specifically thank Doug Hellmann,
> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> Laski for the early feedback that has helped provide some good shape to
> the already discussions.
>
> Some more context on what the happenings:
> We've this in progress spec [1] up for providing context and platform
> for such discussions. I will rephrase it to say that we plan to
> introduce a new 'entity' in the Openstack realm that may be a library or
> a service. Both concepts have trade-offs and the WG wanted to get more
> ideas around such trade-offs from the larger community.
>
> Would you mind to expand this "we" here?


> Service:
> This would entail creating a new project and will introduce managing
> tables for quotas for all the projects that will use this service. For
> example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> be responsible for handling the enforcement, management and DB upgrades
> of the quotas logic for all resources for all three projects. This means
> less pain for projects during the implementation and maintenance phase,
> holistic view of the cloud and almost a guarantee of best practices
> followed (no clutter or guessing around what different projects are
> doing). However, it results into a big dependency; all projects rely on
> this one service for right enforcement, avoiding races (if do not
> incline on implementing some of that in-tree) and DB
> migrations/upgrades. It will be at the core of the cloud and prone to
> attack vectors, bugs and margin of error.
>
> I'd prefer not. As lots of concern raised already ref. latency, extra api
etc.
Based on the unifying the user interface, common api might be desired
option, but it's own service, not so much.


> Library:
> A library could be thought of in two different ways:
> 1) Something that does not deal with backed DB models, provides a
> generic enforcement and management engine. To think ahead a little bit
> it may be a ABC or even a few standard implementation vectors that can
> be imported into a project space. The project will have it's own API for
> quotas and the drivers will enforce different types of logic; per se
> flat quota driver or hierarchical quota driver with custom/project
> specific logic in project tree. Project maintains it's own DB and
> upgrades thereof.
>

Partially decent idea, just the fact annoys me that this is climbing the
tree
arse ahead. The individual API's is perhaps the worst option with common
code for the quotas as each project has their own requirements where the
API might be generalized.

2) A library that has models for DB tables that the project can import
> from. Thus the individual projects will have a handy outline of what the
> tables should look like, implicitly considering the right table values,
> arguments, etc. Project has it's own API and implements drivers in-tree
> by importing this semi-defined structure. Project maintains it's own
> upgrades but will be somewhat influenced by the common repo.
>
> This is really not benefitting anyone. Again each project has their own
requirements for quotas, while the user experience is the one thing we
should try to unify. I have really difficulties to see Zaqar, Nova and
Glance
fitting into the single quota model, while the API interacting with those
could
be similar.

Library would keep things simple for the common repository and sourcing
> of code can be done asynchronously as per project plans and priorities
> without having a strong dependency. On the other hand, there is a
> likelihood of re-implementing similar patterns in different projects
> with individual projects taking responsibility to keep things up to
> date. Attack vectors, bugs and margin of error are project responsibilities
>
> This is the problem I see with oslo approach currently. Originally
intended for
place to collect the common code from projects turning to "enforcing" entity
of code some people thinks should be common and does not fit most.


> Third option is to avoid all of this and simply give guidelines, best
> practices, right packages to each projects to implement quotas in-house.
> Somewhat undesirable at this point, I'd say. But we're all ears!
>

This is probably the best solution without "best practices", again one model
does not suite all, but common concepts, specially on the interacting API
side is desired. _If_ proven that most of the projects would fit to the same
suite, common lib could be built out of the outcome.

>
> Thank you for reading and I 

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Doug Hellmann]

Excerpts from Sean Dague's message of 2016-03-16 06:09:47 -0400:
> On 03/16/2016 05:46 AM, Duncan Thomas wrote:
> > On 16 March 2016 at 09:15, Tim Bell  > > wrote:
> > 
> > Then, there were major reservations from the PTLs at the impacts in
> > terms of
> > latency, ability to reconcile and loss of control (transactions are
> > difficult, transactions
> > across services more so).
> > 
> > 
> > Not just PTLs :-)
> >  
> > 
> > 
> > I would favor a library, at least initially. If we cannot agree on a
> > library, it
> > is unlikely that we can get a service adopted (even if it is desirable).
> > 
> > A library (along the lines of 1 or 2 above) would allow consistent
> > implementation
> > of nested quotas and user quotas. Nested quotas is currently only
> > implemented
> > in Cinder and user quota implementations vary between projects which is
> > confusing.
> > 
> > 
> > It is worth noting that the cinder implementation has been found rather
> > lacking in correctness, atomicity requirements and testing - I wouldn't
> > suggest taking it as anything other than a PoC to be honest. Certainly
> > it should not be cargo-culted into another project in its present state.
> 
> I think a library approach should probably start from scratch, with
> lessons learned from Cinder, but not really copied code, for just that
> reason.
> 
> This is hard code to get right, which is why it's various degrees of
> wrong in every project in OpenStack.
> 
> A common library with it's own db tables and migration train is the only
> way I can imagine this every getting accomplished given the atomicity
> and two phase commit constraints of getting quota on long lived, async
> created resources, with sub resources that also have quota. Definitely
> think that's the nearest term path to victory.

When we talked about this in Paris (I think, all these hotel basements
are starting to look the same), the main issue with the library was how
to tie in db table management with the existing tables owned by the app.
It's not impossible to solve, but we need some thought to happen
around the tools for that. Maybe some of the lessons of incremental
on-demand table updates in nova will help there.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Sean Dague]

On 03/16/2016 08:27 AM, Amrith Kumar wrote:
> Nikhil, thank you for the very timely posting. This is a topic that has
> been discussed quite a bit recently within the Trove team. I've read the
> document you reference as [1] and I have not been part of earlier
> conversations on this subject so I may be missing some context here.
> 
> I feel that the conversation (in [1], in the email thread) has gone to a
> discussion of implementation details (library vs. service, quota
> enforcement engine, interface, ...) when I think there is still some
> ambiguity in my mind about the requirements. What is it that this
> capability will provide and what is the contract implied when a service
> adopts this model.
> 
> For example, consider this case that we see in Trove. In response to a
> user request to create a cluster of databases, Trove must provision
> storage (cinder), compute (nova), networks (let's say neutron), and so
> on. As stated by Boris in his email, it would be ideal if Trove had a
> confirmation from all projects that there was quota available for the
> requests that would be made before the requests actually are made. This
> implies therefore that participating projects (cinder, nova, neutron,
> ...) would have to support some reservations scheme and subsequently
> honor requests based on a reservation. So, I think there's more to this
> than just another library or project, there's an implication for
> projects that wish to participate in this scheme. Or am I wrong in this
> understanding?

I think you have to wind it back further. While Trove wants to get a
consistent lock on quotas in all the projects below it, any single one
of those is massively racy on it's internal quota.

It's totally possible to have nova believe it has enough cpu, memory,
disk, security_groups, floating_ips, instances available for your user,
fail on a reschedule, and end up leaking off chunks of this, and
eventually fail you. So before asking the question about "Can Trove get
a unified quota answer" we have to solve "can the underlying projects
guaruntee consistent quota answers".

There is a giant pile of bugs in Nova about these races, has been
forever, until we solve this in the lower level projects there is no
hope of solving the Trove use case.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Salvatore Orlando]

I'm not sure if this was mentioned already throughout the thread, however
as I've been working a bit on quotas in the past I might have some
additional information:

- Looking at quotas it is worth distinguishing between management (eg::
resource limits per tenant and/or users), and enforcement (eg.: can the
bakery service give me 4 cookies or did I already eat too many?)
  While for the reasons listed throughout this thread the latter should
really happen in the same context where the request is going to served,
quota management instead might be its own service, or however being done in
a common endpoint for all OpenStack resources.
- As far as quota enforcement is concerned, Dims already shared all the
relevant links. You might be already aware that we had a consensus around a
library, but hit a bit of a blocker on the fact that the library should've
introduced db model changes (at the time I devised a massive hack disguised
as abstraction around it). Considering alembic advancements (are we all
using alembic aren't we?) this should not be anymore an issue. I really
would love to have a library that does quota enforcement.
- It has also been raised a good point about securing a chunk of resources
across project, that is also related to John's point about business
quotas... I'm not sure it is necessary, but Blazar [1] kind of achieves
this - even if it was conceived with different purposes.

Salvatore

[1] https://wiki.openstack.org/wiki/Blazar


On 16 March 2016 at 18:27, John Dickinson  wrote:

> There are two types of quotas you may want to enforce in an OpenStack
> project: technical and business.
>
> Technical quotas are things that are hard limits of the system based on
> either actual resources available or protecting the system itself. For
> example, you can't provision a 2TB volume if you only have 1TB of capacity
> available. Similarly, you may want to ratelimit a user to a certain number
> of operations per second in order to keep the system usable by every user.
>
> These sort of quotas should absolutely stay in the realm of each
> individual project. And, for example, if Trove needs to provision a Cinder
> volume but that fails, it's Trove's responsibility for handling that
> elegantly.
>
> Business quotas are different. This is stuff like "a user is allowed to
> provision 1TB of Cinder per Nova compute unit that is provisioned" or "a
> user can provision 1Gb of network capacity per 200TB of data stored in
> Swift". Simpler rules that don't have cross-project dependencies are
> possible too (eg "A user can have no more than 3 compute instances" or "a
> user can have no more than 100k objects or 500TB stored in Swift").
> Oftentimes, these business quotas will be tied in to (or dependent on)
> other product-specific tools like billing or CRM systems.
>
> These business quotas should have a common rules engine in an OpenStack
> deployment. I've long thought that this sort of quota enforcement is an
> authZ decision (i.e. Keystone), but perhaps it's in some other project
> (Congress?). The hard part is that if it's in a central place, that service
> has to be enormously scalable. Specifically, it has to be able to handle
> the aggregate request rate load of every service it is enforcing quotas on.
>
> If we end up with an OpenStack project that is doing centralized business
> quotas, you've got the start of building an ERP system (
> https://en.wikipedia.org/wiki/Enterprise_resource_planning). Frankly, I
> don't think we should be doing that. It's outside of our scope of building
> cloud infrastructure software.
>
> However, we should be all about fixing any problems any individual project
> has about handling technical quotas. That work should stay within its
> respective project. There's no need to consolidate or combine
> project-specific resource management because they happen to all be called
> "quotas".
>
> --John
>
>
>
>
> On 15 Mar 2016, at 23:25, Nikhil Komawar wrote:
>
> > Hello everyone,
> >
> > tl;dr;
> > I'm writing to request some feedback on whether the cross project Quotas
> > work should move ahead as a service or a library or going to a far
> > extent I'd ask should this even be in a common repository, would
> > projects prefer to implement everything from scratch in-tree? Should we
> > limit it to a guideline spec?
> >
> > But before I ask anymore, I want to specifically thank Doug Hellmann,
> > Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> > Laski for the early feedback that has helped provide some good shape to
> > the already discussions.
> >
> > Some more context on what the happenings:
> > We've this in progress spec [1] up for providing context and platform
> > for such discussions. I will rephrase it to say that we plan to
> > introduce a new 'entity' in the Openstack realm that may be a library or
> > a service. Both concepts have trade-offs and the WG wanted to get more
> > ideas around such trade-offs from the larger 

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[John Dickinson]

There are two types of quotas you may want to enforce in an OpenStack project: 
technical and business.

Technical quotas are things that are hard limits of the system based on either 
actual resources available or protecting the system itself. For example, you 
can't provision a 2TB volume if you only have 1TB of capacity available. 
Similarly, you may want to ratelimit a user to a certain number of operations 
per second in order to keep the system usable by every user.

These sort of quotas should absolutely stay in the realm of each individual 
project. And, for example, if Trove needs to provision a Cinder volume but that 
fails, it's Trove's responsibility for handling that elegantly.

Business quotas are different. This is stuff like "a user is allowed to 
provision 1TB of Cinder per Nova compute unit that is provisioned" or "a user 
can provision 1Gb of network capacity per 200TB of data stored in Swift". 
Simpler rules that don't have cross-project dependencies are possible too (eg 
"A user can have no more than 3 compute instances" or "a user can have no more 
than 100k objects or 500TB stored in Swift"). Oftentimes, these business quotas 
will be tied in to (or dependent on) other product-specific tools like billing 
or CRM systems.

These business quotas should have a common rules engine in an OpenStack 
deployment. I've long thought that this sort of quota enforcement is an authZ 
decision (i.e. Keystone), but perhaps it's in some other project (Congress?). 
The hard part is that if it's in a central place, that service has to be 
enormously scalable. Specifically, it has to be able to handle the aggregate 
request rate load of every service it is enforcing quotas on.

If we end up with an OpenStack project that is doing centralized business 
quotas, you've got the start of building an ERP system 
(https://en.wikipedia.org/wiki/Enterprise_resource_planning). Frankly, I don't 
think we should be doing that. It's outside of our scope of building cloud 
infrastructure software.

However, we should be all about fixing any problems any individual project has 
about handling technical quotas. That work should stay within its respective 
project. There's no need to consolidate or combine project-specific resource 
management because they happen to all be called "quotas".

--John




On 15 Mar 2016, at 23:25, Nikhil Komawar wrote:

> Hello everyone,
>
> tl;dr;
> I'm writing to request some feedback on whether the cross project Quotas
> work should move ahead as a service or a library or going to a far
> extent I'd ask should this even be in a common repository, would
> projects prefer to implement everything from scratch in-tree? Should we
> limit it to a guideline spec?
>
> But before I ask anymore, I want to specifically thank Doug Hellmann,
> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> Laski for the early feedback that has helped provide some good shape to
> the already discussions.
>
> Some more context on what the happenings:
> We've this in progress spec [1] up for providing context and platform
> for such discussions. I will rephrase it to say that we plan to
> introduce a new 'entity' in the Openstack realm that may be a library or
> a service. Both concepts have trade-offs and the WG wanted to get more
> ideas around such trade-offs from the larger community.
>
> Service:
> This would entail creating a new project and will introduce managing
> tables for quotas for all the projects that will use this service. For
> example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> be responsible for handling the enforcement, management and DB upgrades
> of the quotas logic for all resources for all three projects. This means
> less pain for projects during the implementation and maintenance phase,
> holistic view of the cloud and almost a guarantee of best practices
> followed (no clutter or guessing around what different projects are
> doing). However, it results into a big dependency; all projects rely on
> this one service for right enforcement, avoiding races (if do not
> incline on implementing some of that in-tree) and DB
> migrations/upgrades. It will be at the core of the cloud and prone to
> attack vectors, bugs and margin of error.
>
> Library:
> A library could be thought of in two different ways:
> 1) Something that does not deal with backed DB models, provides a
> generic enforcement and management engine. To think ahead a little bit
> it may be a ABC or even a few standard implementation vectors that can
> be imported into a project space. The project will have it's own API for
> quotas and the drivers will enforce different types of logic; per se
> flat quota driver or hierarchical quota driver with custom/project
> specific logic in project tree. Project maintains it's own DB and
> upgrades thereof.
> 2) A library that has models for DB tables that the project can import
> from. Thus the individual projects will have a handy outline of 

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Joshua Harlow]

On 03/16/2016 05:22 AM, Julien Danjou wrote:

On Wed, Mar 16 2016, Attila Fazekas wrote:


The quota usage handling MUST happen in the same DB transaction as the
resource record (volume, server..) create/update/delete  .


[…]


We have a transaction capable DB, to help us,
not using it would be lame.


Amen to that.


Double amen!





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Sean Dague]

On 03/16/2016 09:34 AM, gordon chung wrote:
> 
> 
> On 16/03/2016 7:10 AM, Attila Fazekas wrote:
>>
>> NO : For any kind of extra quota service.
>>
>> In other places I saw other reasons for a quota service or similar,
>>   the actual cost of this approach is higher than most people would think so 
>> NO.
>>
>>
>> Maybe Library,
>> But I do not want to see for example the bad pattern used in nova to spread 
>> everywhere.
>>
>> The quota usage handling MUST happen in the same DB transaction as the
>> resource record (volume, server..) create/update/delete  .
>>
>> There is no need for.:
>> - reservation-expirer services or periodic tasks ..
>> - there is no need for quota usage correcter shell scripts or whatever
>> - multiple commits
>>
>>
>> We have a transaction capable DB, to help us,
>> not using it would be lame.
>>
>>
>> [2] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061338.html
> 
> i personally like this reasoning. i also wonder how upgrades are 
> affected because of this decoupling proposal -- does this mean the 
> upgrades of all services would be linked (potentially)?

Not if this was a library that defined tables inside the projects in
question, not in some 3rd db place.

This has to live inside all the upgrade constraints we currently have
(like online data migration in the Nova case), otherwise it's a non starter.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Amrith Kumar]

On 03/16/2016 08:42 AM, Sean Dague wrote:
> On 03/16/2016 08:27 AM, Amrith Kumar wrote:
>> Nikhil, thank you for the very timely posting. This is a topic that has
>> been discussed quite a bit recently within the Trove team. I've read the
>> document you reference as [1] and I have not been part of earlier
>> conversations on this subject so I may be missing some context here.
>>
>> I feel that the conversation (in [1], in the email thread) has gone to a
>> discussion of implementation details (library vs. service, quota
>> enforcement engine, interface, ...) when I think there is still some
>> ambiguity in my mind about the requirements. What is it that this
>> capability will provide and what is the contract implied when a service
>> adopts this model.
>>
>> For example, consider this case that we see in Trove. In response to a
>> user request to create a cluster of databases, Trove must provision
>> storage (cinder), compute (nova), networks (let's say neutron), and so
>> on. As stated by Boris in his email, it would be ideal if Trove had a
>> confirmation from all projects that there was quota available for the
>> requests that would be made before the requests actually are made. This
>> implies therefore that participating projects (cinder, nova, neutron,
>> ...) would have to support some reservations scheme and subsequently
>> honor requests based on a reservation. So, I think there's more to this
>> than just another library or project, there's an implication for
>> projects that wish to participate in this scheme. Or am I wrong in this
>> understanding?
> I think you have to wind it back further. While Trove wants to get a
> consistent lock on quotas in all the projects below it, any single one
> of those is massively racy on it's internal quota.
>
> It's totally possible to have nova believe it has enough cpu, memory,
> disk, security_groups, floating_ips, instances available for your user,
> fail on a reschedule, and end up leaking off chunks of this, and
> eventually fail you. So before asking the question about "Can Trove get
> a unified quota answer" we have to solve "can the underlying projects
> guaruntee consistent quota answers".
>
> There is a giant pile of bugs in Nova about these races, has been
> forever, until we solve this in the lower level projects there is no
> hope of solving the Trove use case.

[amrith] So do I understand that the intent of this project and proposal
is to help address the issues of each individual project, and leave the
cross-project issue for later (or a different project)?

I think it would be beneficial to think of the "unified quota problem"
while designing a solution to the "underlying project problem". It may
even be worthwhile thinking through the design of the unified quota
problem in detail, and implementing the whole thing in phases starting
with the issue of each project. Happy to help with this, let me know how
to volunteer.

>   -Sean
>

--
Amrith Kumar, CTO   | amr...@tesora.com
Tesora, Inc | @amrithkumar
125 CambridgePark Drive, Suite 400  | http://www.tesora.com
Cambridge, MA. 02140| GPG: 0x5e48849a9d21a29b 




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[gordon chung]

On 16/03/2016 7:10 AM, Attila Fazekas wrote:
>
> NO : For any kind of extra quota service.
>
> In other places I saw other reasons for a quota service or similar,
>   the actual cost of this approach is higher than most people would think so 
> NO.
>
>
> Maybe Library,
> But I do not want to see for example the bad pattern used in nova to spread 
> everywhere.
>
> The quota usage handling MUST happen in the same DB transaction as the
> resource record (volume, server..) create/update/delete  .
>
> There is no need for.:
> - reservation-expirer services or periodic tasks ..
> - there is no need for quota usage correcter shell scripts or whatever
> - multiple commits
>
>
> We have a transaction capable DB, to help us,
> not using it would be lame.
>
>
> [2] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061338.html

i personally like this reasoning. i also wonder how upgrades are 
affected because of this decoupling proposal -- does this mean the 
upgrades of all services would be linked (potentially)?

cheers,
-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Amrith Kumar]

Nikhil, thank you for the very timely posting. This is a topic that has
been discussed quite a bit recently within the Trove team. I've read the
document you reference as [1] and I have not been part of earlier
conversations on this subject so I may be missing some context here.

I feel that the conversation (in [1], in the email thread) has gone to a
discussion of implementation details (library vs. service, quota
enforcement engine, interface, ...) when I think there is still some
ambiguity in my mind about the requirements. What is it that this
capability will provide and what is the contract implied when a service
adopts this model.

For example, consider this case that we see in Trove. In response to a
user request to create a cluster of databases, Trove must provision
storage (cinder), compute (nova), networks (let's say neutron), and so
on. As stated by Boris in his email, it would be ideal if Trove had a
confirmation from all projects that there was quota available for the
requests that would be made before the requests actually are made. This
implies therefore that participating projects (cinder, nova, neutron,
...) would have to support some reservations scheme and subsequently
honor requests based on a reservation. So, I think there's more to this
than just another library or project, there's an implication for
projects that wish to participate in this scheme. Or am I wrong in this
understanding?

Several have observed that we have a 'transaction capable DB to help
us'. Let me go further and say that we have a distributed transaction
capable DB (yes, if you squint just the right way, MySQL can do 2phase
commit) and we should leverage that to the hilt.

But, I'd like to get caught up on the requirements that this capability
is to provide before we get into implementations, so if there are any
descriptions of those requirements, I'd appreciate getting pointers to that.

Thanks,

-amrith


[1] https://review.openstack.org/#/c/284454/

--
Amrith Kumar, CTO   | amr...@tesora.com
Tesora, Inc | @amrithkumar
125 CambridgePark Drive, Suite 400  | http://www.tesora.com
Cambridge, MA. 02140| GPG: 0x5e48849a9d21a29b 

On 03/16/2016 02:25 AM, Nikhil Komawar wrote:
> Hello everyone,
>
> tl;dr;
> I'm writing to request some feedback on whether the cross project Quotas
> work should move ahead as a service or a library or going to a far
> extent I'd ask should this even be in a common repository, would
> projects prefer to implement everything from scratch in-tree? Should we
> limit it to a guideline spec?
>
> But before I ask anymore, I want to specifically thank Doug Hellmann,
> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> Laski for the early feedback that has helped provide some good shape to
> the already discussions.
>
> Some more context on what the happenings:
> We've this in progress spec [1] up for providing context and platform
> for such discussions. I will rephrase it to say that we plan to
> introduce a new 'entity' in the Openstack realm that may be a library or
> a service. Both concepts have trade-offs and the WG wanted to get more
> ideas around such trade-offs from the larger community.
>
> Service:
> This would entail creating a new project and will introduce managing
> tables for quotas for all the projects that will use this service. For
> example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> be responsible for handling the enforcement, management and DB upgrades
> of the quotas logic for all resources for all three projects. This means
> less pain for projects during the implementation and maintenance phase,
> holistic view of the cloud and almost a guarantee of best practices
> followed (no clutter or guessing around what different projects are
> doing). However, it results into a big dependency; all projects rely on
> this one service for right enforcement, avoiding races (if do not
> incline on implementing some of that in-tree) and DB
> migrations/upgrades. It will be at the core of the cloud and prone to
> attack vectors, bugs and margin of error.
>
> Library:
> A library could be thought of in two different ways:
> 1) Something that does not deal with backed DB models, provides a
> generic enforcement and management engine. To think ahead a little bit
> it may be a ABC or even a few standard implementation vectors that can
> be imported into a project space. The project will have it's own API for
> quotas and the drivers will enforce different types of logic; per se
> flat quota driver or hierarchical quota driver with custom/project
> specific logic in project tree. Project maintains it's own DB and
> upgrades thereof.
> 2) A library that has models for DB tables that the project can import
> from. Thus the individual projects will have a handy outline of what the
> tables should look like, implicitly considering the right table values,
> arguments, etc. Project has it's own API 

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[John Garbutt]

On 16 March 2016 at 10:09, Sean Dague  wrote:
> On 03/16/2016 05:46 AM, Duncan Thomas wrote:
>> On 16 March 2016 at 09:15, Tim Bell > > wrote:
>>
>> Then, there were major reservations from the PTLs at the impacts in
>> terms of
>> latency, ability to reconcile and loss of control (transactions are
>> difficult, transactions
>> across services more so).
>>
>>
>> Not just PTLs :-)
>>
>>
>> 
>> I would favor a library, at least initially. If we cannot agree on a
>> library, it
>> is unlikely that we can get a service adopted (even if it is desirable).
>>
>> A library (along the lines of 1 or 2 above) would allow consistent
>> implementation
>> of nested quotas and user quotas. Nested quotas is currently only
>> implemented
>> in Cinder and user quota implementations vary between projects which is
>> confusing.
>>
>>
>> It is worth noting that the cinder implementation has been found rather
>> lacking in correctness, atomicity requirements and testing - I wouldn't
>> suggest taking it as anything other than a PoC to be honest. Certainly
>> it should not be cargo-culted into another project in its present state.
>
> I think a library approach should probably start from scratch, with
> lessons learned from Cinder, but not really copied code, for just that
> reason.
>
> This is hard code to get right, which is why it's various degrees of
> wrong in every project in OpenStack.

+1

> A common library with it's own db tables and migration train is the only
> way I can imagine this every getting accomplished given the atomicity
> and two phase commit constraints of getting quota on long lived, async
> created resources, with sub resources that also have quota. Definitely
> think that's the nearest term path to victory.

As an aside, I believe Neutron have had some recent success with a
quota re-write, although I have not dug into that.

In Nova we have been discussing a new approach to the DB storage of
the quota, loosely similar to the resource provider claims system we
are currently adding into the scheduler:
https://review.openstack.org/#/c/182445/4/specs/backlog/approved/quotas-reimagined.rst

We also discussed the idea of dropping a lot quota reservation
complexity. Rather than worrying if the async create server succeeds,
we could just commit the quota as soon as the server is recorded in
the db. This means an failed build consumes quota until the instance
is deleted, but that seems like a good trade off. Similarly, resize up
could just consume quota (until delete or revert resize), rather than
waiting for resize confirm.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Adam Young]

On 03/16/2016 10:04 AM, Davanum Srinivas wrote:

To complete the context:
https://review.openstack.org/#/c/132127/
https://etherpad.openstack.org/p/kilo-oslo-common-quota-library (from
https://wiki.openstack.org/wiki/Design_Summit/Kilo/Etherpads)

-- Dims

On Wed, Mar 16, 2016 at 9:53 AM, Doug Hellmann  wrote:

Excerpts from Sean Dague's message of 2016-03-16 06:09:47 -0400:

On 03/16/2016 05:46 AM, Duncan Thomas wrote:

On 16 March 2016 at 09:15, Tim Bell > wrote:

 Then, there were major reservations from the PTLs at the impacts in
 terms of
 latency, ability to reconcile and loss of control (transactions are
 difficult, transactions
 across services more so).


Not just PTLs :-)


 
 I would favor a library, at least initially. If we cannot agree on a
 library, it
 is unlikely that we can get a service adopted (even if it is desirable).

 A library (along the lines of 1 or 2 above) would allow consistent
 implementation
 of nested quotas and user quotas. Nested quotas is currently only
 implemented
 in Cinder and user quota implementations vary between projects which is
 confusing.


It is worth noting that the cinder implementation has been found rather
lacking in correctness, atomicity requirements and testing - I wouldn't
suggest taking it as anything other than a PoC to be honest. Certainly
it should not be cargo-culted into another project in its present state.

I think a library approach should probably start from scratch, with
lessons learned from Cinder, but not really copied code, for just that
reason.

This is hard code to get right, which is why it's various degrees of
wrong in every project in OpenStack.

A common library with it's own db tables and migration train is the only
way I can imagine this every getting accomplished given the atomicity
and two phase commit constraints of getting quota on long lived, async
created resources, with sub resources that also have quota. Definitely
think that's the nearest term path to victory.

When we talked about this in Paris (I think, all these hotel basements
are starting to look the same), the main issue with the library was how
to tie in db table management with the existing tables owned by the app.
It's not impossible to solve, but we need some thought to happen
around the tools for that. Maybe some of the lessons of incremental
on-demand table updates in nova will help there.

Doug


Assuming we are working with SQL Alchemy and Migrations, I would suggest 
that the quota table be put into its own table namespace, separate from 
the rest of the application.  We did this for a while with Keystone 
extensions, in order to allow them to avoid revision number conflicts as 
reviews came in.


The Migrations table looks like this:
desc migrate_version;
+-+--+--+-+-+---+
| Field   | Type | Null | Key | Default | Extra |
+-+--+--+-+-+---+
| repository_id   | varchar(250) | NO   | PRI | NULL|   |
| repository_path | text | YES  | | NULL|   |
| version | int(11)  | YES  | | NULL|   |
+-+--+--+-+-+---+

And you can see there is a repository_path variable that lets that vary.

Keystone has only:


 select * from migrate_version;
+---+---+-+
| repository_id | 
repository_path   | 
version |

+---+---+-+
| keystone  | 
/usr/lib/python2.7/site-packages/keystone/common/sql/migrate_repo |  
94 |

+---+---+-+


Note that there is one of these tables per Service database, so even if 
Keystoen and Nova share a mysql instance, they have their own copy of 
this table.










__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Davanum Srinivas]

To complete the context:
https://review.openstack.org/#/c/132127/
https://etherpad.openstack.org/p/kilo-oslo-common-quota-library (from
https://wiki.openstack.org/wiki/Design_Summit/Kilo/Etherpads)

-- Dims

On Wed, Mar 16, 2016 at 9:53 AM, Doug Hellmann  wrote:
> Excerpts from Sean Dague's message of 2016-03-16 06:09:47 -0400:
>> On 03/16/2016 05:46 AM, Duncan Thomas wrote:
>> > On 16 March 2016 at 09:15, Tim Bell > > > wrote:
>> >
>> > Then, there were major reservations from the PTLs at the impacts in
>> > terms of
>> > latency, ability to reconcile and loss of control (transactions are
>> > difficult, transactions
>> > across services more so).
>> >
>> >
>> > Not just PTLs :-)
>> >
>> >
>> > 
>> > I would favor a library, at least initially. If we cannot agree on a
>> > library, it
>> > is unlikely that we can get a service adopted (even if it is 
>> > desirable).
>> >
>> > A library (along the lines of 1 or 2 above) would allow consistent
>> > implementation
>> > of nested quotas and user quotas. Nested quotas is currently only
>> > implemented
>> > in Cinder and user quota implementations vary between projects which is
>> > confusing.
>> >
>> >
>> > It is worth noting that the cinder implementation has been found rather
>> > lacking in correctness, atomicity requirements and testing - I wouldn't
>> > suggest taking it as anything other than a PoC to be honest. Certainly
>> > it should not be cargo-culted into another project in its present state.
>>
>> I think a library approach should probably start from scratch, with
>> lessons learned from Cinder, but not really copied code, for just that
>> reason.
>>
>> This is hard code to get right, which is why it's various degrees of
>> wrong in every project in OpenStack.
>>
>> A common library with it's own db tables and migration train is the only
>> way I can imagine this every getting accomplished given the atomicity
>> and two phase commit constraints of getting quota on long lived, async
>> created resources, with sub resources that also have quota. Definitely
>> think that's the nearest term path to victory.
>
> When we talked about this in Paris (I think, all these hotel basements
> are starting to look the same), the main issue with the library was how
> to tie in db table management with the existing tables owned by the app.
> It's not impossible to solve, but we need some thought to happen
> around the tools for that. Maybe some of the lessons of incremental
> on-demand table updates in nova will help there.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Julien Danjou]

On Wed, Mar 16 2016, Attila Fazekas wrote:

> The quota usage handling MUST happen in the same DB transaction as the 
> resource record (volume, server..) create/update/delete  .

[…]

> We have a transaction capable DB, to help us,
> not using it would be lame.

Amen to that.

-- 
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Sean McGinnis]

On Wed, Mar 16, 2016 at 07:10:35AM -0400, Attila Fazekas wrote:
> 
> NO : For any kind of extra quota service.
> 
> In other places I saw other reasons for a quota service or similar,
>  the actual cost of this approach is higher than most people would think so 
> NO.
> 

I have to agree, there is a lot of hidden cost with making this its own
service, both on the development and end user side of things.

> 
> Maybe Library,
> But I do not want to see for example the bad pattern used in nova to spread 
> everywhere.
> 
> The quota usage handling MUST happen in the same DB transaction as the 
> resource record (volume, server..) create/update/delete  .
> 
> There is no need for.:
> - reservation-expirer services or periodic tasks ..
> - there is no need for quota usage correcter shell scripts or whatever
> - multiple commits
> 
> 
> We have a transaction capable DB, to help us,
> not using it would be lame.

I would much rather see a library that at least ensures our handling of
quotas is consistent across projects. We've implemented some things and
had some issues. Nova has implemented some things and had some issues.
Let's get this all in one place so we can all benefit from lessons
learned and work through these issues.

I'm sure there will still be problems. But I think it ends up being a
better user experience even if there are some quirks in the library. At
least they will be consistent quirks across all the projects instead of
needing to understand each project's own set of misbehavior.

> 
> 
> [2] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061338.html
> 
> - Original Message -
> > From: "Nikhil Komawar" <nik.koma...@gmail.com>
> > To: "OpenStack Development Mailing List" <openstack-dev@lists.openstack.org>
> > Sent: Wednesday, March 16, 2016 7:25:26 AM
> > Subject: [openstack-dev] [cross-project] [all] Quotas -- service vs. library
> > 
> > Hello everyone,
> > 
> > tl;dr;
> > I'm writing to request some feedback on whether the cross project Quotas
> > work should move ahead as a service or a library or going to a far
> > extent I'd ask should this even be in a common repository, would
> > projects prefer to implement everything from scratch in-tree? Should we
> > limit it to a guideline spec?
> > 
> > But before I ask anymore, I want to specifically thank Doug Hellmann,
> > Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> > Laski for the early feedback that has helped provide some good shape to
> > the already discussions.
> > 
> > Some more context on what the happenings:
> > We've this in progress spec [1] up for providing context and platform
> > for such discussions. I will rephrase it to say that we plan to
> > introduce a new 'entity' in the Openstack realm that may be a library or
> > a service. Both concepts have trade-offs and the WG wanted to get more
> > ideas around such trade-offs from the larger community.
> > 
> > Service:
> > This would entail creating a new project and will introduce managing
> > tables for quotas for all the projects that will use this service. For
> > example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> > be responsible for handling the enforcement, management and DB upgrades
> > of the quotas logic for all resources for all three projects. This means
> > less pain for projects during the implementation and maintenance phase,
> > holistic view of the cloud and almost a guarantee of best practices
> > followed (no clutter or guessing around what different projects are
> > doing). However, it results into a big dependency; all projects rely on
> > this one service for right enforcement, avoiding races (if do not
> > incline on implementing some of that in-tree) and DB
> > migrations/upgrades. It will be at the core of the cloud and prone to
> > attack vectors, bugs and margin of error.
> > 
> > Library:
> > A library could be thought of in two different ways:
> > 1) Something that does not deal with backed DB models, provides a
> > generic enforcement and management engine. To think ahead a little bit
> > it may be a ABC or even a few standard implementation vectors that can
> > be imported into a project space. The project will have it's own API for
> > quotas and the drivers will enforce different types of logic; per se
> > flat quota driver or hierarchical quota driver with custom/project
> > specific logic in project tree. Project maintains it's own DB and
> > upgrades thereof.
> > 2) A library that has models for DB tables that the project can import
> &

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Joshua Harlow]

On 03/16/2016 05:42 AM, Sean Dague wrote:

On 03/16/2016 08:27 AM, Amrith Kumar wrote:

Nikhil, thank you for the very timely posting. This is a topic that has
been discussed quite a bit recently within the Trove team. I've read the
document you reference as [1] and I have not been part of earlier
conversations on this subject so I may be missing some context here.

I feel that the conversation (in [1], in the email thread) has gone to a
discussion of implementation details (library vs. service, quota
enforcement engine, interface, ...) when I think there is still some
ambiguity in my mind about the requirements. What is it that this
capability will provide and what is the contract implied when a service
adopts this model.

For example, consider this case that we see in Trove. In response to a
user request to create a cluster of databases, Trove must provision
storage (cinder), compute (nova), networks (let's say neutron), and so
on. As stated by Boris in his email, it would be ideal if Trove had a
confirmation from all projects that there was quota available for the
requests that would be made before the requests actually are made. This
implies therefore that participating projects (cinder, nova, neutron,
...) would have to support some reservations scheme and subsequently
honor requests based on a reservation. So, I think there's more to this
than just another library or project, there's an implication for
projects that wish to participate in this scheme. Or am I wrong in this
understanding?


I think you have to wind it back further. While Trove wants to get a
consistent lock on quotas in all the projects below it, any single one
of those is massively racy on it's internal quota.

It's totally possible to have nova believe it has enough cpu, memory,
disk, security_groups, floating_ips, instances available for your user,
fail on a reschedule, and end up leaking off chunks of this, and
eventually fail you. So before asking the question about "Can Trove get
a unified quota answer" we have to solve "can the underlying projects
guaruntee consistent quota answers".

There is a giant pile of bugs in Nova about these races, has been
forever, until we solve this in the lower level projects there is no
hope of solving the Trove use case.


+1 I think/thought the goal was more of to solve all the above bugs 
first by doing something/thinking a little different and figuring out 
how to make this quota service (or library) a reality (finally!!)


Then sure, of course, we can work on those other problems as well 
(obviously we should try to design a initial solution that should not 
make solving those kinds of problems impossible).


-josh



-Sean



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Attila Fazekas]

NO : For any kind of extra quota service.

In other places I saw other reasons for a quota service or similar,
 the actual cost of this approach is higher than most people would think so NO.


Maybe Library,
But I do not want to see for example the bad pattern used in nova to spread 
everywhere.

The quota usage handling MUST happen in the same DB transaction as the 
resource record (volume, server..) create/update/delete  .

There is no need for.:
- reservation-expirer services or periodic tasks ..
- there is no need for quota usage correcter shell scripts or whatever
- multiple commits


We have a transaction capable DB, to help us,
not using it would be lame.


[2] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061338.html

- Original Message -
> From: "Nikhil Komawar" <nik.koma...@gmail.com>
> To: "OpenStack Development Mailing List" <openstack-dev@lists.openstack.org>
> Sent: Wednesday, March 16, 2016 7:25:26 AM
> Subject: [openstack-dev] [cross-project] [all] Quotas -- service vs. library
> 
> Hello everyone,
> 
> tl;dr;
> I'm writing to request some feedback on whether the cross project Quotas
> work should move ahead as a service or a library or going to a far
> extent I'd ask should this even be in a common repository, would
> projects prefer to implement everything from scratch in-tree? Should we
> limit it to a guideline spec?
> 
> But before I ask anymore, I want to specifically thank Doug Hellmann,
> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> Laski for the early feedback that has helped provide some good shape to
> the already discussions.
> 
> Some more context on what the happenings:
> We've this in progress spec [1] up for providing context and platform
> for such discussions. I will rephrase it to say that we plan to
> introduce a new 'entity' in the Openstack realm that may be a library or
> a service. Both concepts have trade-offs and the WG wanted to get more
> ideas around such trade-offs from the larger community.
> 
> Service:
> This would entail creating a new project and will introduce managing
> tables for quotas for all the projects that will use this service. For
> example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> be responsible for handling the enforcement, management and DB upgrades
> of the quotas logic for all resources for all three projects. This means
> less pain for projects during the implementation and maintenance phase,
> holistic view of the cloud and almost a guarantee of best practices
> followed (no clutter or guessing around what different projects are
> doing). However, it results into a big dependency; all projects rely on
> this one service for right enforcement, avoiding races (if do not
> incline on implementing some of that in-tree) and DB
> migrations/upgrades. It will be at the core of the cloud and prone to
> attack vectors, bugs and margin of error.
> 
> Library:
> A library could be thought of in two different ways:
> 1) Something that does not deal with backed DB models, provides a
> generic enforcement and management engine. To think ahead a little bit
> it may be a ABC or even a few standard implementation vectors that can
> be imported into a project space. The project will have it's own API for
> quotas and the drivers will enforce different types of logic; per se
> flat quota driver or hierarchical quota driver with custom/project
> specific logic in project tree. Project maintains it's own DB and
> upgrades thereof.
> 2) A library that has models for DB tables that the project can import
> from. Thus the individual projects will have a handy outline of what the
> tables should look like, implicitly considering the right table values,
> arguments, etc. Project has it's own API and implements drivers in-tree
> by importing this semi-defined structure. Project maintains it's own
> upgrades but will be somewhat influenced by the common repo.
> 
> Library would keep things simple for the common repository and sourcing
> of code can be done asynchronously as per project plans and priorities
> without having a strong dependency. On the other hand, there is a
> likelihood of re-implementing similar patterns in different projects
> with individual projects taking responsibility to keep things up to
> date. Attack vectors, bugs and margin of error are project responsibilities
> 
> Third option is to avoid all of this and simply give guidelines, best
> practices, right packages to each projects to implement quotas in-house.
> Somewhat undesirable at this point, I'd say. But we're all ears!
> 
> Thank you for reading and I anticipate more feedback.
> 
> [1] https://review.openstack.org/#/c/284454/
> 
> --
> 
> Thanks,
>

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Sean Dague]

On 03/16/2016 05:46 AM, Duncan Thomas wrote:
> On 16 March 2016 at 09:15, Tim Bell  > wrote:
> 
> Then, there were major reservations from the PTLs at the impacts in
> terms of
> latency, ability to reconcile and loss of control (transactions are
> difficult, transactions
> across services more so).
> 
> 
> Not just PTLs :-)
>  
> 
> 
> I would favor a library, at least initially. If we cannot agree on a
> library, it
> is unlikely that we can get a service adopted (even if it is desirable).
> 
> A library (along the lines of 1 or 2 above) would allow consistent
> implementation
> of nested quotas and user quotas. Nested quotas is currently only
> implemented
> in Cinder and user quota implementations vary between projects which is
> confusing.
> 
> 
> It is worth noting that the cinder implementation has been found rather
> lacking in correctness, atomicity requirements and testing - I wouldn't
> suggest taking it as anything other than a PoC to be honest. Certainly
> it should not be cargo-culted into another project in its present state.

I think a library approach should probably start from scratch, with
lessons learned from Cinder, but not really copied code, for just that
reason.

This is hard code to get right, which is why it's various degrees of
wrong in every project in OpenStack.

A common library with it's own db tables and migration train is the only
way I can imagine this every getting accomplished given the atomicity
and two phase commit constraints of getting quota on long lived, async
created resources, with sub resources that also have quota. Definitely
think that's the nearest term path to victory.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Duncan Thomas]

On 16 March 2016 at 09:15, Tim Bell  wrote:

Then, there were major reservations from the PTLs at the impacts in terms of
> latency, ability to reconcile and loss of control (transactions are
> difficult, transactions
> across services more so).
>
>
Not just PTLs :-)


> 
> I would favor a library, at least initially. If we cannot agree on a
> library, it
> is unlikely that we can get a service adopted (even if it is desirable).
>
> A library (along the lines of 1 or 2 above) would allow consistent
> implementation
> of nested quotas and user quotas. Nested quotas is currently only
> implemented
> in Cinder and user quota implementations vary between projects which is
> confusing.


It is worth noting that the cinder implementation has been found rather
lacking in correctness, atomicity requirements and testing - I wouldn't
suggest taking it as anything other than a PoC to be honest. Certainly it
should not be cargo-culted into another project in its present state.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Boris Pavlovic]

Nikhil,

Thank you for rising this question.

IMHO quotas should be moved into separated services (this is right micro
services approach).

It will make a lot of things simpler:
1) this removes a lot of logic/code from projects
2) cross project atomic quotas reservation (will be possible)
   (e.g. if we would like to reserver all required quotas, before running
heat stack)
3) it will have better UX (you can change projects quotas from one place
and unified)
4) simpler migrations for the projects (we don't need to maintain db
migrations for each project)
just imagine change in quotas lib that requires db migrations, we will
need to run amount of projects migrations.


Best regards,
Boris Pavlovic


On Tue, Mar 15, 2016 at 11:25 PM, Nikhil Komawar 
wrote:

> Hello everyone,
>
> tl;dr;
> I'm writing to request some feedback on whether the cross project Quotas
> work should move ahead as a service or a library or going to a far
> extent I'd ask should this even be in a common repository, would
> projects prefer to implement everything from scratch in-tree? Should we
> limit it to a guideline spec?
>
> But before I ask anymore, I want to specifically thank Doug Hellmann,
> Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
> Laski for the early feedback that has helped provide some good shape to
> the already discussions.
>
> Some more context on what the happenings:
> We've this in progress spec [1] up for providing context and platform
> for such discussions. I will rephrase it to say that we plan to
> introduce a new 'entity' in the Openstack realm that may be a library or
> a service. Both concepts have trade-offs and the WG wanted to get more
> ideas around such trade-offs from the larger community.
>
> Service:
> This would entail creating a new project and will introduce managing
> tables for quotas for all the projects that will use this service. For
> example if Nova, Glance, and Cinder decide to use it, this 'entity' will
> be responsible for handling the enforcement, management and DB upgrades
> of the quotas logic for all resources for all three projects. This means
> less pain for projects during the implementation and maintenance phase,
> holistic view of the cloud and almost a guarantee of best practices
> followed (no clutter or guessing around what different projects are
> doing). However, it results into a big dependency; all projects rely on
> this one service for right enforcement, avoiding races (if do not
> incline on implementing some of that in-tree) and DB
> migrations/upgrades. It will be at the core of the cloud and prone to
> attack vectors, bugs and margin of error.
>
> Library:
> A library could be thought of in two different ways:
> 1) Something that does not deal with backed DB models, provides a
> generic enforcement and management engine. To think ahead a little bit
> it may be a ABC or even a few standard implementation vectors that can
> be imported into a project space. The project will have it's own API for
> quotas and the drivers will enforce different types of logic; per se
> flat quota driver or hierarchical quota driver with custom/project
> specific logic in project tree. Project maintains it's own DB and
> upgrades thereof.
> 2) A library that has models for DB tables that the project can import
> from. Thus the individual projects will have a handy outline of what the
> tables should look like, implicitly considering the right table values,
> arguments, etc. Project has it's own API and implements drivers in-tree
> by importing this semi-defined structure. Project maintains it's own
> upgrades but will be somewhat influenced by the common repo.
>
> Library would keep things simple for the common repository and sourcing
> of code can be done asynchronously as per project plans and priorities
> without having a strong dependency. On the other hand, there is a
> likelihood of re-implementing similar patterns in different projects
> with individual projects taking responsibility to keep things up to
> date. Attack vectors, bugs and margin of error are project responsibilities
>
> Third option is to avoid all of this and simply give guidelines, best
> practices, right packages to each projects to implement quotas in-house.
> Somewhat undesirable at this point, I'd say. But we're all ears!
>
> Thank you for reading and I anticipate more feedback.
>
> [1] https://review.openstack.org/#/c/284454/
>
> --
>
> Thanks,
> Nikhil
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Tim Bell]

On 16/03/16 07:25, "Nikhil Komawar"  wrote:



>Hello everyone,
>
>tl;dr;
>I'm writing to request some feedback on whether the cross project Quotas
>work should move ahead as a service or a library or going to a far
>extent I'd ask should this even be in a common repository, would
>projects prefer to implement everything from scratch in-tree? Should we
>limit it to a guideline spec?
>
>But before I ask anymore, I want to specifically thank Doug Hellmann,
>Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
>Laski for the early feedback that has helped provide some good shape to
>the already discussions.
>
>Some more context on what the happenings:
>We've this in progress spec [1] up for providing context and platform
>for such discussions. I will rephrase it to say that we plan to
>introduce a new 'entity' in the Openstack realm that may be a library or
>a service. Both concepts have trade-offs and the WG wanted to get more
>ideas around such trade-offs from the larger community.
>
>Service:
>This would entail creating a new project and will introduce managing
>tables for quotas for all the projects that will use this service. For
>example if Nova, Glance, and Cinder decide to use it, this 'entity' will
>be responsible for handling the enforcement, management and DB upgrades
>of the quotas logic for all resources for all three projects. This means
>less pain for projects during the implementation and maintenance phase,
>holistic view of the cloud and almost a guarantee of best practices
>followed (no clutter or guessing around what different projects are
>doing). However, it results into a big dependency; all projects rely on
>this one service for right enforcement, avoiding races (if do not
>incline on implementing some of that in-tree) and DB
>migrations/upgrades. It will be at the core of the cloud and prone to
>attack vectors, bugs and margin of error.

This has been proposed a number of times in the past with projects such as Boson
(https://wiki.openstack.org/wiki/Boson) and an extended discussion at one of the
summits (I think it was San Diego).

Then, there were major reservations from the PTLs at the impacts in terms of
latency, ability to reconcile and loss of control (transactions are difficult, 
transactions
across services more so).

>Library:
>A library could be thought of in two different ways:
>1) Something that does not deal with backed DB models, provides a
>generic enforcement and management engine. To think ahead a little bit
>it may be a ABC or even a few standard implementation vectors that can
>be imported into a project space. The project will have it's own API for
>quotas and the drivers will enforce different types of logic; per se
>flat quota driver or hierarchical quota driver with custom/project
>specific logic in project tree. Project maintains it's own DB and
>upgrades thereof.
>2) A library that has models for DB tables that the project can import
>from. Thus the individual projects will have a handy outline of what the
>tables should look like, implicitly considering the right table values,
>arguments, etc. Project has it's own API and implements drivers in-tree
>by importing this semi-defined structure. Project maintains it's own
>upgrades but will be somewhat influenced by the common repo.
>
>Library would keep things simple for the common repository and sourcing
>of code can be done asynchronously as per project plans and priorities
>without having a strong dependency. On the other hand, there is a
>likelihood of re-implementing similar patterns in different projects
>with individual projects taking responsibility to keep things up to
>date. Attack vectors, bugs and margin of error are project responsibilities
>
>Third option is to avoid all of this and simply give guidelines, best
>practices, right packages to each projects to implement quotas in-house.
>Somewhat undesirable at this point, I'd say. But we're all ears!

I would favor a library, at least initially. If we cannot agree on a library, it
is unlikely that we can get a service adopted (even if it is desirable).

A library (along the lines of 1 or 2 above) would allow consistent 
implementation
of nested quotas and user quotas. Nested quotas is currently only implemented
in Cinder and user quota implementations vary between projects which is 
confusing.

Now that we have Oslo (there was no similar structure when it was first 
discussed),
we have the possibility to implement these concepts in a consistent way across
OpenStack and give a better user experience as a result.

Tim

>
>Thank you for reading and I anticipate more feedback.
>
>[1] https://review.openstack.org/#/c/284454/
>
>-- 
>
>Thanks,
>Nikhil
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

[openstack-dev] [cross-project] [all] Quotas -- service vs. library

[Nikhil Komawar]

Hello everyone,

tl;dr;
I'm writing to request some feedback on whether the cross project Quotas
work should move ahead as a service or a library or going to a far
extent I'd ask should this even be in a common repository, would
projects prefer to implement everything from scratch in-tree? Should we
limit it to a guideline spec?

But before I ask anymore, I want to specifically thank Doug Hellmann,
Joshua Harlow, Davanum Srinivas, Sean Dague, Sean McGinnis and  Andrew
Laski for the early feedback that has helped provide some good shape to
the already discussions.

Some more context on what the happenings:
We've this in progress spec [1] up for providing context and platform
for such discussions. I will rephrase it to say that we plan to
introduce a new 'entity' in the Openstack realm that may be a library or
a service. Both concepts have trade-offs and the WG wanted to get more
ideas around such trade-offs from the larger community.

Service:
This would entail creating a new project and will introduce managing
tables for quotas for all the projects that will use this service. For
example if Nova, Glance, and Cinder decide to use it, this 'entity' will
be responsible for handling the enforcement, management and DB upgrades
of the quotas logic for all resources for all three projects. This means
less pain for projects during the implementation and maintenance phase,
holistic view of the cloud and almost a guarantee of best practices
followed (no clutter or guessing around what different projects are
doing). However, it results into a big dependency; all projects rely on
this one service for right enforcement, avoiding races (if do not
incline on implementing some of that in-tree) and DB
migrations/upgrades. It will be at the core of the cloud and prone to
attack vectors, bugs and margin of error.

Library:
A library could be thought of in two different ways:
1) Something that does not deal with backed DB models, provides a
generic enforcement and management engine. To think ahead a little bit
it may be a ABC or even a few standard implementation vectors that can
be imported into a project space. The project will have it's own API for
quotas and the drivers will enforce different types of logic; per se
flat quota driver or hierarchical quota driver with custom/project
specific logic in project tree. Project maintains it's own DB and
upgrades thereof.
2) A library that has models for DB tables that the project can import
from. Thus the individual projects will have a handy outline of what the
tables should look like, implicitly considering the right table values,
arguments, etc. Project has it's own API and implements drivers in-tree
by importing this semi-defined structure. Project maintains it's own
upgrades but will be somewhat influenced by the common repo.

Library would keep things simple for the common repository and sourcing
of code can be done asynchronously as per project plans and priorities
without having a strong dependency. On the other hand, there is a
likelihood of re-implementing similar patterns in different projects
with individual projects taking responsibility to keep things up to
date. Attack vectors, bugs and margin of error are project responsibilities

Third option is to avoid all of this and simply give guidelines, best
practices, right packages to each projects to implement quotas in-house.
Somewhat undesirable at this point, I'd say. But we're all ears!

Thank you for reading and I anticipate more feedback.

[1] https://review.openstack.org/#/c/284454/

-- 

Thanks,
Nikhil


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev