On 06/08/15 08:20, Grasza, Grzegorz wrote:
-----Original Message-----
From: Zane Bitter [mailto:zbit...@redhat.com]
Sent: Thursday, 6 August, 2015 2:57
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Heat] RPC API versioning
We've been talking about this since before summit without much consensus.
I think a large part of the problem is that very few people have deep
knowledge of both Heat and Versioned Objects. However, I think we are at a
point where we should be able to settle on an approach at least for the API<-
engine RPC interface. I've been talking to Dan Smith about the Nova team's
plan for upgrades, which goes something like this:
* Specify a max RPC API version in the config file
* In the RPC client lib, add code to handle versions as far back as the previous
release
* The operator rolls out the updated code, keeping the existing config file
with the pin to the previous release's RPC version
* Once all services are upgraded, the operator rolls out a new config file
shifting the pin
* The backwards compat code to handle release N-1 is removed in the N+1
release
This is, I believe, sufficient to solve our entire problem.
Specifically, we have no need for an indirection API that rebroadcasts
messages that are too new (since that can't happen with pinning) and no
need for Versioned Objects in the RPC layer. (Versioned objects for the DB
are still critical, and we are very much better off for all the hard work that
Michal and others have put into them. Thanks!)
What is the use of versioned objects outside of RPC?
I've written some documentation for Oslo VO and helped in introducing them in
Heat.
As I understand it, the only use cases for VO are
* to serialize objects to dicts with version information when they are sent
over RPC
* handle version dependent code inside the objects (instead of scattering it
around the codebase)
This is what we currently have them for AIUI - so that we can do updates
to the DB schema without downtime. The VO abstracts the
version-dependent parts in a single place, so the rest of the code can
continue to work even with an old DB schema. This means that we can roll
out new code and only update the schema once it is all running.
* provide an object oriented and transparent access to the resources
represented by the objects to services which don't have direct access to that
resource (via RPC) - the indirection API
The last point was not yet discussed in Heat as far as I know, but the
indirection API also contains an interface for backporting objects, which is
something that is currently only used in Nova, and as you say, doesn't have a
use when version pinning is in place.
The nature of Heat's RPC API is that it is effectively user-facing - the
heat-api
process is essentially a thin proxy between ReST and RPC. We already have a
translation layer between the internal representation(s) of objects and the
user-facing representation, in the form of heat.engine.api, and the RPC API is
firmly on the user-facing side. The requirements for the content of these
messages are actually much stricter than anything we need for RPC API
stability, since they need to remain compatible not just with heat-api but
with heatclient - and we have *zero* control over when that gets upgraded.
Despite that, we've managed quite nicely for ~3 years without breaking
changes afaik.
Versioned Objects is a great way of retaining control when you need to share
internal data structures between processes. Fortunately the architecture of
Heat makes that unnecessary. That was a good design decision. We are not
going to reverse that design decision in order to use Versioned Objects. (In
the interest of making sure everyone uses their time productively, perhaps I
should clarify that to: "your patch is subject to -2 on sight if it introduces
internal engine data structures to heat-api/heat-cfn-api".)
Hopefully I've convinced you of the sufficiency of this plan for the API<-
engine interface specifically. If anyone disagrees, let them speak now, &c.
I don't understand - what is the distinction between internal and external data
structures?
Internal data structures are just whatever it's most convenient for the
application to work with. They're subject to change/refactoring at any time.
External data structures are ones that have an explicit stability
guarantee, in this case because they're part of the user interface.
From what I understand, versioned objects were introduced in Heat to represent
objects which are sent over RPC between Heat services.
I believe you have been misinformed. All of the versioned objects that
have been created so far are representations of tables in the database.
However, we never send these over RPC: pre-VO it would obviously have
been dumb to couple the user interface to the representation in the
database, so we just didn't do that and therefore that was never a
motivation for creating those VOs. And given that we already have a
functioning abstraction layer in the engine, I don't think it makes
sense to rewrite it and move it to a different process now.
I think there is still a case that could be made for a different approach to the
RPC API for convergence, which is engine->engine and
(probably) doesn't yet have a formal translation layer of the same kind.
At a minimum, obviously, we should do the same stuff listed above (though I
don't think we need to declare that interface stable until the first release
where we enable convergence by default).
I agree this could be a good use case for VO.
There's probably places where versioned objects could benefit us. For
example, when we trigger a check on a resource we pass it a bundle of data
containing all the attributes and IDs it might need from resources it depends
on. It definitely makes sense to me that that bundle would be a Versioned
Object. (In fact, that data gets stored in the DB - as SyncPoint in the
prototype - so we wouldn't even need to create a new object type. This
seems like a clear win.)
What I do NOT want to do is to e.g. replace the resource_id of the resource
to check with a versioned object containing the DB representation of a
resource. In most cases that doesn't save a DB lookup, it just moves it to the
other end of the RPC call. And as long as primary keys are unique, this can
never create a compatibility problem :)
Agreed. In a case it saves additional DB lookups though, sending an object can
be a good option.
Yeah, I agree that it could be in that case. I don't think that's the
case, but it would be nice to have the flexibility to adopt that in the
future if it turns out that it is. Do you think your proposed solution
(i.e. the indirection API mentioned below) would make it easier to e.g.
swap the resource key for a versioned object in a future version of the
message?
Grzegorz mentioned in https://review.openstack.org/#/c/196670/ the
possibility of using the indirection API to handle changes to the message
format - I guess this is an alternative to the version pinning approach
mentioned by Dan? I'm currently leaning toward thinking that this is
unnecessary, but I'm not sure I understand it completely so I'd need to see a
patch to form an opinion. I'm wide open to persuasion on this part. However
I'd be opposed to this *replacing* (rather than
supplementing) the version pinning behaviour described above - the whole
retransmission of too-new messages thing sounds like it could have any
number of pathological corner cases and I'd prefer to avoid it.
The only difference in my proposed solution is that the "if version == 'x.x'"
code would be constrained to the version object implementation (inside the heat/objects
directory), and the rest of the code could use the object properties and methods without
being concerned about versioning.
OK, thanks for clarifying.
cheers,
Zane.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
