Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-25 Thread Chris Dent
There's a review in progress for a generic event format for PaaS-services which is a move with the right spirit: allow various services to join the the notification party without needing special handlers. See: https://review.openstack.org/#/c/101967/ -- Chris Dent tw:@anticdent freenode:cdent h

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-17 Thread Chris Dent
On Tue, 15 Jul 2014, Sandy Walsh wrote: This looks like a particular schema for one event-type (let's say "foo.sample"). It's hard to extrapolate this one schema to a generic set of common metadata applicable to all events. Really the only common stuff we can agree on is the stuff already there

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-17 Thread Chris Dent
On Tue, 15 Jul 2014, Mark McLoughlin wrote: So you're proposing that all payloads should contain something like: [...] a class, type, id, value, unit and a space to put additional metadata. That's the gist of it, but I'm only presenting that as a way to get somebody else to point out what's wr

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-16 Thread Sandy Walsh
On 7/11/2014 6:08 AM, Chris Dent wrote: > On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: > >> The data format that Ironic will send was part of the spec proposed >> and could have been reviewed. I think there's still time to change it >> tho, if you have a better format talk to Haomeng which is th

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-15 Thread Sandy Walsh
On 7/15/2014 3:51 AM, Mark McLoughlin wrote: > On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote: >> On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: >> >>> The data format that Ironic will send was part of the spec proposed >>> and could have been reviewed. I think there's still time to change it

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-15 Thread Mark McLoughlin
On Thu, 2014-07-10 at 16:21 -0400, Eoghan Glynn wrote: > > > One of the issues that has been raised in the recent discussions with > > > the QA team about branchless Tempest relates to some legacy defects > > > in the OpenStack notification system. > > > > Got links to specifics? I thought the con

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-14 Thread Mark McLoughlin
On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote: > On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: > > > The data format that Ironic will send was part of the spec proposed > > and could have been reviewed. I think there's still time to change it > > tho, if you have a better format talk to Ha

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-14 Thread Joe Gordon
On Thu, Jul 10, 2014 at 1:48 AM, Eoghan Glynn wrote: > > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? > > Folks, > > One of the issues that has been raised in the recent discussions with > the QA team about branchless Tempest relates to some

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-14 Thread Eoghan Glynn
> > So what we need to figure out is how exactly this common structure can be > > accommodated without reverting back to what Sandy called the "wild west" > > in another post. > > I got the impression that "wild west" is what we've already got > (within the payload)? Yeah, exactly, that was my

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-14 Thread Chris Dent
On Sat, 12 Jul 2014, Eoghan Glynn wrote: So what we need to figure out is how exactly this common structure can be accommodated without reverting back to what Sandy called the "wild west" in another post. I got the impression that "wild west" is what we've already got (within the payload)? F

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-12 Thread Eoghan Glynn
> > A notification of compute.instance.create.start is naturally going to > > carry different types of data than a volume.snapshot.delete.end for > > example, but of course we'd seek to accommodate that difference within > > a generic structure as far as possible. > > Is it going to carry differ

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Chris Dent
On Fri, 11 Jul 2014, Eoghan Glynn wrote: A notification of compute.instance.create.start is naturally going to carry different types of data than a volume.snapshot.delete.end for example, but of course we'd seek to accommodate that difference within a generic structure as far as possible. Is i

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Sandy Walsh
On 7/11/2014 6:04 AM, Chris Dent wrote: > On Fri, 11 Jul 2014, Eoghan Glynn wrote: > >> But I guess you're suggesting not only that we version/schematize >> individual notification payloads, but that we do so in a way that's >> global across event types and emitters? > That's partially correct. I'm

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Sandy Walsh
On 7/10/2014 12:10 PM, Chris Dent wrote: > On Thu, 10 Jul 2014, Julien Danjou wrote: > >> My initial plan was to leverage a library like voluptuous to do schema >> based validation on the sender side. That would allow for receiver to >> introspect schema and know the data structure to expect. I did

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Eoghan Glynn
> > But I guess you're suggesting not only that we version/schematize > > individual notification payloads, but that we do so in a way that's > > global across event types and emitters? > > That's partially correct. I'm suggesting the we consider standardizing a > general format for notification

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Chris Dent
On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if you have a better format talk to Haomeng which is the guys responsible for that work in Ironic and see

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Chris Dent
On Fri, 11 Jul 2014, Eoghan Glynn wrote: But I guess you're suggesting not only that we version/schematize individual notification payloads, but that we do so in a way that's global across event types and emitters? That's partially correct. I'm suggesting the we consider standardizing a genera

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Lucas Alvares Gomes
> I just started the code for processing of notifications from Ironic. > Conceptually they are the same as notifications from Nova but the > actual form of the payload is completely different. This means I have to > write a different processor for that payload. And now so does StackTach > if they w

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Eoghan Glynn
> tl;dr: Having a minimum payload standard enables more and more robust > services on both sides of notification bus. Yes, that's exactly the point of this thread. > We do not have a standard format for the payload. I think we should > (more on that below). Again, such standardization is exact

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Chris Dent
tl;dr: Having a minimum payload standard enables more and more robust services on both sides of notification bus. Expanded earnest optimism below. On Thu, 10 Jul 2014, Eoghan Glynn wrote: So it's the standardization of notification payload formats that's being proposed. Generally that payloa

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Matt Dietz
-Original Message- From: Eoghan Glynn Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Thursday, July 10, 2014 at 3:30 PM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [all] Treatin

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
> The original implementation of the notification system was never intended > to be the *final* implementation. I think we all identified the need for > versioning several years ago. As for backwards compatibility, I think the > version field itself, in whatever form it takes, should be optional.

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
> From my perspective, the requirement is to be able to have a consistent and > predictable format for notifications that are being sent from all services. > This means: > 1. a set of required fields that all events contain and have consistent > meaning > 2. a set of optional fields, you don’t ha

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
> > One of the issues that has been raised in the recent discussions with > > the QA team about branchless Tempest relates to some legacy defects > > in the OpenStack notification system. > > Got links to specifics? I thought the consensus was that there was a > contract here which we need to mai

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
> It's not clear to me in this discussion what it is that is being > versioned, contracted or standardized. Well the statement in the original mail was pretty explicit: "versioned notification payloads to protect consumers from breaking changes in payload format" So it's the standardization

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
> Glad to see this coming up again. I've been pushing that since the > Icehouse summit in Hong-Kong, where we had a Oslo session about that > subject and where everybody nod about my initial plan. :) > > I've created a blueprint and started a very primitive implementation > back then: > > htt

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
> Thanks for dusting this off. Versioning and published schemas for > notifications are important to the StackTach team. It would be nice to > get this resolved. We're happy to help out. Great! > > A great outcome would include some or all of the following: > > > > 1. more complete in-tree te

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Sandy Walsh
On 7/10/2014 2:59 PM, Daniel Dyer wrote: > From my perspective, the requirement is to be able to have a consistent and > predictable format for notifications that are being sent from all services. > This means: > 1. a set of required fields that all events contain and have consistent > meaning >

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Daniel Dyer
>From my perspective, the requirement is to be able to have a consistent and >predictable format for notifications that are being sent from all services. >This means: 1. a set of required fields that all events contain and have consistent meaning 2. a set of optional fields, you don’t have to inc

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Matt Dietz
-Original Message- From: Sandy Walsh Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Thursday, July 10, 2014 at 9:31 AM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [all] Treatin

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Mark McLoughlin
On Thu, 2014-07-10 at 04:48 -0400, Eoghan Glynn wrote: > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? > > Folks, > > One of the issues that has been raised in the recent discussions with > the QA team about branchless Tempest relates to some

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Chris Dent
On Thu, 10 Jul 2014, Julien Danjou wrote: My initial plan was to leverage a library like voluptuous to do schema based validation on the sender side. That would allow for receiver to introspect schema and know the data structure to expect. I didn't think deeply on how to handle versioning, but t

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Sandy Walsh
On 7/10/2014 5:52 AM, Eoghan Glynn wrote: > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? Thanks for dusting this off. Versioning and published schemas for notifications are important to the StackTach team. It would be nice to get this resolve

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Matt Riedemann
On 7/10/2014 3:48 AM, Eoghan Glynn wrote: TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Julien Danjou
On Thu, Jul 10 2014, Eoghan Glynn wrote: Hi Eoghan, > So the purpose of this thread is simply to get a read on the appetite > in the community for such an effort. At the least it would require: > > * trashing out the details in say a cross-project-track session at >the K* summit > > * buy-i

[openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the OpenStack notification system. Now, I don't p