Hi Chris,
Looking at your requirements, seems my solution (see attached email) is pretty
much aligned. What I am trying to propose is
1. One root domain as owner of "virtual cloud". Logically linked to "n" leaf
domains.
2. All leaf domains falls under admin boundary of "virtual cloud" owner.
3. No sharing of resources at project level, that will keep the authorization
model simple.
4. No sharing of resources at domain level either.
5. Hierarchy or admin boundary will be totally governed by roles.
This way we can setup a true virtual cloud/Reseller/wholesale model.
Thoughts?
Thanks,
Arvind
-----Original Message-----
From: Chris Behrens [mailto:cbehr...@codestud.com]
Sent: Wednesday, February 05, 2014 1:27 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [keystone][nova] Re: Hierarchicical Multitenancy
Discussion
Hi Vish,
I'm jumping in slightly late on this, but I also have an interest in this. I'm
going to preface this by saying that I have not read this whole thread yet, so
I apologize if I repeat things, say anything that is addressed by previous
posts, or doesn't jive with what you're looking for. :) But what you describe
below sounds like exactly a use case I'd come up with.
Essentially I want another level above project_id. Depending on the exact use
case, you could name it 'wholesale_id' or 'reseller_id'...and yeah, 'org_id'
fits in with your example. :) I think that I had decided I'd call it 'domain'
to be more generic, especially after seeing keystone had a domain concept.
Your idea below (prefixing the project_id) is exactly one way I thought of
doing this to be least intrusive. I, however, thought that this would not be
efficient. So, I was thinking about proposing that we add 'domain' to all of
our models. But that limits your hierarchy and I don't necessarily like that.
:) So I think that if the queries are truly indexed as you say below, you have
a pretty good approach. The one issue that comes into mind is that if there's
any chance of collision. For example, if project ids (or orgs) could contain a
'.', then '.' as a delimiter won't work.
My requirements could be summed up pretty well by thinking of this as 'virtual
clouds within a cloud'. Deploy a single cloud infrastructure that could look
like many multiple clouds. 'domain' would be the key into each different
virtual cloud. Accessing one virtual cloud doesn't reveal any details about
another virtual cloud.
What this means is:
1) domain 'a' cannot see instances (or resources in general) in domain 'b'. It
doesn't matter if domain 'a' and domain 'b' share the same tenant ID. If you
act with the API on behalf of domain 'a', you cannot see your instances in
domain 'b'.
2) Flavors per domain. domain 'a' can have different flavors than domain 'b'.
3) Images per domain. domain 'a' could see different images than domain 'b'.
4) Quotas and quota limits per domain. your instances in domain 'a' don't count
against quotas in domain 'b'.
5) Go as far as using different config values depending on what domain you're
using. This one is fun. :)
etc.
I'm not sure if you were looking to go that far or not. :) But I think that our
ideas are close enough, if not exact, that we can achieve both of our goals
with the same implementation.
I'd love to be involved with this. I am not sure that I currently have the time
to help with implementation, however.
- Chris
On Feb 3, 2014, at 1:58 PM, Vishvananda Ishaya <vishvana...@gmail.com> wrote:
> Hello Again!
>
> At the meeting last week we discussed some options around getting true
> multitenancy in nova. The use case that we are trying to support can be
> described as follows:
>
> "Martha, the owner of ProductionIT provides it services to multiple
> Enterprise clients. She would like to offer cloud services to Joe at
> WidgetMaster, and Sam at SuperDevShop. Joe is a Development Manager for
> WidgetMaster and he has multiple QA and Development teams with many users.
> Joe needs the ability create users, projects, and quotas, as well as the
> ability to list and delete resources across WidgetMaster. Martha needs to be
> able to set the quotas for both WidgetMaster and SuperDevShop; manage users,
> projects, and objects across the entire system; and set quotas for the client
> companies as a whole. She also needs to ensure that Joe can't see or mess
> with anything owned by Sam."
>
> As per the plan I outlined in the meeting I have implemented a
> Proof-of-Concept that would allow me to see what changes were required in
> nova to get scoped tenancy working. I used a simple approach of faking out
> heirarchy by prepending the id of the larger scope to the id of the smaller
> scope. Keystone uses uuids internally, but for ease of explanation I will
> pretend like it is using the name. I think we can all agree that
> 'orga.projecta' is more readable than
> 'b04f9ea01a9944ac903526885a2666dec45674c5c2c6463dad3c0cb9d7b8a6d8'.
>
> The code basically creates the following five projects:
>
> orga
> orga.projecta
> orga.projectb
> orgb
> orgb.projecta
>
> I then modified nova to replace everywhere where it searches or limits policy
> by project_id to do a prefix match. This means that someone using project
> 'orga' should be able to list/delete instances in orga, orga.projecta, and
> orga.projectb.
>
> You can find the code here:
>
>
> https://github.com/vishvananda/devstack/commit/10f727ce39ef4275b613201
> ae1ec7655bd79dd5f
> https://github.com/vishvananda/nova/commit/ae4de19560b0a3718efaffb6c20
> 5c7a3c372412f
>
> Keeping in mind that this is a prototype, but I'm hoping to come to some kind
> of consensus as to whether this is a reasonable approach. I've compiled a
> list of pros and cons.
>
> Pros:
>
> * Very easy to understand
> * Minimal changes to nova
> * Good performance in db (prefix matching uses indexes)
> * Could be extended to cover more complex scenarios like multiple
> owners or multiple scopes
>
> Cons:
>
> * Nova has no map of the hierarchy
> * Moving projects would require updates to ownership inside of nova
> * Complex scenarios involving delegation of roles may be a bad fit
> * Database upgrade to hierarchy could be tricky
>
> If this seems like a reasonable set of tradeoffs, there are a few things that
> need to be done inside of nova bring this to a complete solution:
>
> * Prefix matching needs to go into oslo.policy
> * Should the tenant_id returned by the api reflect the full 'orga.projecta',
> or just the child 'projecta' or match the scope: i.e. the first if you are
> authenticated to orga and the second if you are authenticated to the project?
> * Possible migrations for existing project_id fields
> * Use a different field for passing ownership scope instead of
> overloading project_id
> * Figure out how nested quotas should work
> * Look for other bugs relating to scoping
>
> Also, we need to decide how keystone should construct and pass this
> information to the services. The obvious case that could be supported today
> would be to allow a single level of hierarchy using domains. For example, if
> domains are active, keystone could pass domain.project_id for
> ownership_scope. This could be controversial because potentially domains are
> just for grouping users and shouldn't be applied to projects.
>
> I think the real value of this approach would be to allow nested projects
> with role inheritance. When keystone is creating the token, it could walk the
> tree of parent projects, construct the set of roles, and construct the
> ownership_scope as it walks to the root of the tree.
>
> Finally, similar fixes will need to be made in the other projects to bring
> this to a complete solution.
>
> Please feel free to respond with any input, and we will be having another
> Hierarchical Multitenancy Meeting on Friday at 1600 UTC to discuss.
>
> Vish
>
> On Jan 28, 2014, at 10:35 AM, Vishvananda Ishaya <vishvana...@gmail.com>
> wrote:
>
>> Hi Everyone,
>>
>> I apologize for the obtuse title, but there isn't a better succinct term to
>> describe what is needed. OpenStack has no support for multiple owners of
>> objects. This means that a variety of private cloud use cases are simply not
>> supported. Specifically, objects in the system can only be managed on the
>> tenant level or globally.
>>
>> The key use case here is to delegate administration rights for a group of
>> tenants to a specific user/role. There is something in Keystone called a
>> "domain" which supports part of this functionality, but without support from
>> all of the projects, this concept is pretty useless.
>>
>> In IRC today I had a brief discussion about how we could address this. I
>> have put some details and a straw man up here:
>>
>> https://wiki.openstack.org/wiki/HierarchicalMultitenancy
>>
>> I would like to discuss this strawman and organize a group of people to get
>> actual work done by having an irc meeting this Friday at 1600UTC. I know
>> this time is probably a bit tough for Europe, so if we decide we need a
>> regular meeting to discuss progress then we can vote on a better time for
>> this meeting.
>>
>> https://wiki.openstack.org/wiki/Meetings#Hierarchical_Multitenancy_Me
>> eting
>>
>> Please note that this is going to be an active team that produces code. We
>> will *NOT* spend a lot of time debating approaches, and instead focus on
>> making something that works and learning as we go. The output of this team
>> will be a MultiTenant devstack install that actually works, so that we can
>> ensure the features we are adding to each project work together.
>>
>> Vish
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--- Begin Message ---
Hi Vish,
I am sorry as I am proposing just a solution approach below but no code so far.
### Problem and Requirement ###
As per the problem description it seems to me that "Martha, the owner of
ProductionIT" is not a cloud provider (correct me if wrong) and she uses
someone else cloud infrastructure (like public cloud) to provide services to
multiple Enterprise clients.
After reading further it seems to me that we want different administrative
boundaries and isolation among them, so that "Joe can manage/mess-up resource
for WidgetMaster and Sam for SuperDevShop but not each other" at the same time
"Martha should be allowed to manage resources from both".
### Requirements based on my understanding ###
1. Joe can manage/mess-up resource for WidgetMaster and Sam for SuperDevShop
but not each other.
2. Martha should be able to manage resources from both.
### Solution ### (*This solution is backed-up by production working
implementation*)
In a nutshell, We need ability to setup hierarchicical administrative
boundaries within a cloud deployment, so that multiple service providers (like
Martha) can be created and have administrative privilege across multiple
domains which falls under their administrative boundary.
Note: There will be only one level of hierarchy as multi level hierarchy is
complex to implement and does not perform well. Give that,
"Martha/ProductionIT" will be at root of hierarchy, Joe and Sam will stand at
leaf of the hierarchy.
(Solution for Req#1) Keystone has concept of domains which is nothing but a
notion of an administrative boundary where admin of one domain is allowed to
manage resources within a specific domain but not across domains, provided
correct policy is in place. This is already in place so there will be no change
needed to support Req #1.
(Solution for Req#2)
We can extend the notion of domains further to define a root (parent/super)
domain and leaf (child/sub) domains, one set of root and leaf domains will
define a hierarchicical administrative boundary. Glue between root and leafs
will be different roles, that way "Matha" can become "NovaAdmin" (or any other
role) in WidgetMaster or SuperDevShop.
### Pros ###
Complete IAM based solution.
More logically fits in hierarchicical model.
Absolutely no (or minimal) changes to services (Nova, Swift ....)
### Cons ###
Most of the changes is needed in Keystone and its data model (Domain, Roles).
Some change required in OSLO policy engine for policy evaluation.
Service (Nova, Swift ....) may have define new roles.
Let me know if I am not making sense here or have any questions.
Arvind
-----Original Message-----
From: Vishvananda Ishaya [mailto:vishvana...@gmail.com]
Sent: Monday, February 03, 2014 2:58 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [keystone][nova] Re: Hierarchicical Multitenancy
Discussion
Hello Again!
At the meeting last week we discussed some options around getting true
multitenancy in nova. The use case that we are trying to support can be
described as follows:
"Martha, the owner of ProductionIT provides it services to multiple Enterprise
clients. She would like to offer cloud services to Joe at WidgetMaster, and Sam
at SuperDevShop. Joe is a Development Manager for WidgetMaster and he has
multiple QA and Development teams with many users. Joe needs the ability create
users, projects, and quotas, as well as the ability to list and delete
resources across WidgetMaster. Martha needs to be able to set the quotas for
both WidgetMaster and SuperDevShop; manage users, projects, and objects across
the entire system; and set quotas for the client companies as a whole. She also
needs to ensure that Joe can't see or mess with anything owned by Sam."
As per the plan I outlined in the meeting I have implemented a Proof-of-Concept
that would allow me to see what changes were required in nova to get scoped
tenancy working. I used a simple approach of faking out heirarchy by prepending
the id of the larger scope to the id of the smaller scope. Keystone uses uuids
internally, but for ease of explanation I will pretend like it is using the
name. I think we can all agree that 'orga.projecta' is more readable than
'b04f9ea01a9944ac903526885a2666dec45674c5c2c6463dad3c0cb9d7b8a6d8'.
The code basically creates the following five projects:
orga
orga.projecta
orga.projectb
orgb
orgb.projecta
I then modified nova to replace everywhere where it searches or limits policy
by project_id to do a prefix match. This means that someone using project
'orga' should be able to list/delete instances in orga, orga.projecta, and
orga.projectb.
You can find the code here:
https://github.com/vishvananda/devstack/commit/10f727ce39ef4275b613201ae1ec7655bd79dd5f
https://github.com/vishvananda/nova/commit/ae4de19560b0a3718efaffb6c205c7a3c372412f
Keeping in mind that this is a prototype, but I'm hoping to come to some kind
of consensus as to whether this is a reasonable approach. I've compiled a list
of pros and cons.
Pros:
* Very easy to understand
* Minimal changes to nova
* Good performance in db (prefix matching uses indexes)
* Could be extended to cover more complex scenarios like multiple owners or
multiple scopes
Cons:
* Nova has no map of the hierarchy
* Moving projects would require updates to ownership inside of nova
* Complex scenarios involving delegation of roles may be a bad fit
* Database upgrade to hierarchy could be tricky
If this seems like a reasonable set of tradeoffs, there are a few things that
need to be done inside of nova bring this to a complete solution:
* Prefix matching needs to go into oslo.policy
* Should the tenant_id returned by the api reflect the full 'orga.projecta',
or just the child 'projecta' or match the scope: i.e. the first if you are
authenticated to orga and the second if you are authenticated to the project?
* Possible migrations for existing project_id fields
* Use a different field for passing ownership scope instead of overloading
project_id
* Figure out how nested quotas should work
* Look for other bugs relating to scoping
Also, we need to decide how keystone should construct and pass this information
to the services. The obvious case that could be supported today would be to
allow a single level of hierarchy using domains. For example, if domains are
active, keystone could pass domain.project_id for ownership_scope. This could
be controversial because potentially domains are just for grouping users and
shouldn't be applied to projects.
I think the real value of this approach would be to allow nested projects with
role inheritance. When keystone is creating the token, it could walk the tree
of parent projects, construct the set of roles, and construct the
ownership_scope as it walks to the root of the tree.
Finally, similar fixes will need to be made in the other projects to bring this
to a complete solution.
Please feel free to respond with any input, and we will be having another
Hierarchical Multitenancy Meeting on Friday at 1600 UTC to discuss.
Vish
On Jan 28, 2014, at 10:35 AM, Vishvananda Ishaya <vishvana...@gmail.com> wrote:
> Hi Everyone,
>
> I apologize for the obtuse title, but there isn't a better succinct term to
> describe what is needed. OpenStack has no support for multiple owners of
> objects. This means that a variety of private cloud use cases are simply not
> supported. Specifically, objects in the system can only be managed on the
> tenant level or globally.
>
> The key use case here is to delegate administration rights for a group of
> tenants to a specific user/role. There is something in Keystone called a
> "domain" which supports part of this functionality, but without support from
> all of the projects, this concept is pretty useless.
>
> In IRC today I had a brief discussion about how we could address this. I have
> put some details and a straw man up here:
>
> https://wiki.openstack.org/wiki/HierarchicalMultitenancy
>
> I would like to discuss this strawman and organize a group of people to get
> actual work done by having an irc meeting this Friday at 1600UTC. I know this
> time is probably a bit tough for Europe, so if we decide we need a regular
> meeting to discuss progress then we can vote on a better time for this
> meeting.
>
> https://wiki.openstack.org/wiki/Meetings#Hierarchical_Multitenancy_Meeting
>
> Please note that this is going to be an active team that produces code. We
> will *NOT* spend a lot of time debating approaches, and instead focus on
> making something that works and learning as we go. The output of this team
> will be a MultiTenant devstack install that actually works, so that we can
> ensure the features we are adding to each project work together.
>
> Vish
--- End Message ---
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
