On 06/04/2015 05:49 PM, Morgan Fainberg wrote:
Hi Everyone!
I've been reading through this thread and have had some conversations
along the side and wanted to jump in to distill out what I think are
the key points we are trying to address here. I'm going to outline
about 4 items that seem to make sense to me regarding the evolution of
policy. I also want to say that the "notification" that something has
changed from the defaults in a way that may cause odd behavior to the
side (the warning Sean was outlining); we can keep the desire to have
those types of warnings for operators down the line (nothing that is
being proposed here or what I'm going to outline will make it more or
less difficult to add the functionality later on). This is not to say
we wouldn't provide validation of an override, but a subjective "this
is a problematic policy configuration" doesn't need to be directly
part of this conversation today (it can happen once we know what the
model of policy looks like going forward).
1. The first thing that I'm hearing from the conversation (this is
based upon Sean's proposal) is that we already trust the individual
projects to know the enforcement profile for their resources. It seems
like the project should be authoritative on what that enforcement
should look like. Handing off the enforcement definition to Keystone
is the wrong direction. I really like the concept of defining within
Nova the default policy that nova works with (we do this already
today, and I don't want to require the nova team to come to Keystone
to make changes to the policy down the line). The Projects are trusted
to know what the enforcement points are and trusted to distribute a
basic profile of enforcement.
To the end that the enforcement definition is handled by the
individual projects, making it something that is more than a "blob of
text" also makes a lot of sense. A code-like model that is easier to
understand for the developers that are implementing enforcement would
be useful. The key pieces are that this code-like-construct must be
able to be serialized out into the "common" format.
The policy file format is JSON, which is the standard for all of the
APIs in OpenStack thus far.
Second, this code-construct is just the basic level of "truth", the
idea is that the dynamic policy will provide the overrides - and
*everything* can be overridden. The code-like construct will also aid
in profiling/testing the base/defaults (and then the dynamic policy
overrides) without having to standup the entire stack. We can enable
base functionality testing / validation and then the more integrated
testing with the full stack (in different environments). This will
enable more accurate and better base policy development by the teams
we already trust to build the enforcement for a given project (e.g. Nova).
We already have this. It is the default policy.json that each project
keeps up to date.
Please don't suggest putting annotations on the coder and running a
preprocessor. That way leads to madness. I see no reason to have a
team write policy in Python and then serialize to JSON.
--
The real current problem we have is this: On a given API, we don't know
where to look for the project ID and (almost all) policy needs to be
enforced on the project scope. What is required is for the base
repository to have a document of how to set up the scoping for the call
(token.proiject.id must match fetched_object.tenant_id), and we could
mark that as "dangerous to change". What I would not want is to have
the project hardcode the Role required. Perhaps the API indicates one
of two levels: Admin vs Member, on an API, indicating the expected
consumer of the API. However, the current Policy file format represents
this sufficiently. We just need the Nova team to stay on top of this
for Nova, and the other teams for their projects.
What I would love to be able to get from Nova is "this api will end up
calling these apis in Glance, Cinder, and neutron." So we can properly
delegate.
2. We will need a way to handle the bi-directional notifications for
policy changes. This would encompass when a project is restarted and
has a new code-policy construct and how that gets to Keystone. We also
need to ensure that changes to the overrides are pushed down to the
projects. This is not an easy canned solution today, but I am sure we
can solve it. Likely this is tied to Keystone Middleware or something
similar (I believe code is already in the works to this end).
I appreciate that Nova is growing, but I would not expect huge amounts
of new policy code with each updtate...at this point, new policy would
be "deny until there is a new rule to handle it" and "upload these new
rules". Upgrading code is already part of an organizations deployment
stategy, and trying to build in an additional two way notification
system is more than we should be doing here.
If Nova feels that each micro version needs its own delta to the policy
file, it can produce them, but I would be wary of just blanket
overriding what the operators have in production.
--
3. The HA-Proxy mode of deployment with projects that can handle
no-downtime upgrades mean that we need to add in versioning into the
policy structures. The policy files for a "Kilo" vintage of Nova may
(likely) will be incompatible with "Liberty" nova. This means we
cannot assume that policy can be centralized easily even for a
specific grouping of api-services running as a single endpoint. This
becomes an even more important mechanism as we move towards more and
more services with microversioned APIs. It means it is totally
reasonable to upgrade 1 or 2 nova APIs behind an HA Proxy since the
new APIs will handle the old microversion of the API.
This does not correspond with what I have seen from the policy files
thus far. If microversions are going to require significantly different
policy, they need to fetch rules from the policy file. APIs themselves
do not change that much. When Keystone went from V2 to V3, it underwent
a radical rewriting of the strucutre of the data, but those are handled
by different API calls, and thus different policy calls.
This leads to needing policy to likewise be versioned. This also means
that only the service can be authoritative with the base-policy
construct. This means whatever tool we use for handling the overrides
on the Keystone side will need to be aware of policy versions as well.
Having Keystone side being exclusively authoratative for the entire
policy makes development, testing, and understanding of policy harder.
This is another case of the project itself should be in control of the
base policy definition.
Again, this does not correspond with what I have seen in the current
policy files. We've been locked to static policy for as long as we've
had policy in OpenStack. We finally have the opportunity to move
forward. A default rule of "deny until we have a viable policy" makes
far more sense than trying to automatically merge in new policies when
code updates.
Since the policy files replace existing rules, it would make more sense
for Nova to put a new micro-version policy file with just the new rules
into its repository. We could make one option on the database be that
when an update comes in, reject it if it attempts to overwrite any
existing rules, and use that for loading deltas. That way, we can append
new policy.json files to existing ones without breaking an existing
deployment.
Yes, it means that you can't redefine how policy works on a given API,
but you should not do that anyway. That breaks people's custom policy,
and that is not kind to operators.
--
4. As a note that came out of the conversation I had with Sean, we
should look at no longer making the policy definition for an API keyed
on the intern-method name of a project. While "instance_create" is
relatively descriptive, there are many other API calls that you cannot
really know what changing the policy will do without trying it. Sean
and I were discussing moving towards supplying a representation of the
URI path instead of "image_create". This is something that consumers
and deployers of OpenStack will be more familiar with. It also
eliminates some of the mapping needed to know what the URI of "image
create" is when utilized in the Horizon context (asking Keystone "what
can be done with authorization XXX").
We would all like that.Keep in mind, however that there is nothing that
requires policy only be enforced once on an API call, or that policy
cannot be called and enforced deep within the code. A project may
decide that it wants to enforce policy at the Database layer, and make
sure that all requests that attempt to read a specific record pass a
policy check. We should not prevent them from doing so.
--
All of the conversation in this thread and in IRC has been good. But I
think we're at the point where we need to start setting a clear
direction on the policy bits. All the recommendations have had merit
and bring to the table interesting perspectives.
I'd like to actually start making progress on the Dynamic policy
implementation. The current approach is broken and a security risk.I
think the current approach covers Sean's concerns without writing policy
in Python code.
Lets start defining the next steps (this is not going to be easy) so
we can start really working on the foundation we need to make all of
this a reality.
--Morgan
On Thu, Jun 4, 2015 at 12:00 PM, Adam Young <ayo...@redhat.com
<mailto:ayo...@redhat.com>> wrote:
On 06/04/2015 01:16 PM, Sean Dague wrote:
It gets overwritten by the central store.
And you are wrong, that gives me what I want, because we can
emit a
WARNING in the logs if the patch is something crazy. The
operators will
see it, and be able to fix it later.
I'm not trying to prevent people from changing their policy in
crazy
ways. I'm trying to build in some safety net where we can
detect it's
kind of a bad idea and emit that information a place that
Operators can
see and sort out later, instead of pulling their hair out.
But you can only do that if you have encoded what's the
default, plus
annotations about ways that changing the default are unwise.
When would you expect this warning to be emitted, and to whom? I
think you have the right idea, but I suggest that the appropriate
time to give that warning would be back when the policy is
written, which would be under the scope of the Database-Driven
policy management. I would think that, if a user changes a
policy, it would go to a staged state, not deployed immediately,
and at that point, we'd want a check to run. That check would be
what told the author they did something unexpected. Waiting until
the policy hits the server is probably too late.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
