----- Original Message ----- > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Should we target it for Kilo? It does not seem right to allow it > slipping into the next release while we know there are operators > relying on the feature.
Of course, this will be fixed for Kilo. This is the immediate fix, which should be merged right away: https://review.openstack.org/#/c/163676/ I pushed a patch to support servers with multiple namespaces to Oslo messaging: https://review.openstack.org/#/c/163673/ Once that gains momentum I'll send a patch to Neutron that re-enables namespaces for RPC servers (Along with the null namespace) and enable fallbacks for clients. > > On 03/11/2015 08:42 PM, Assaf Muller wrote: > > I've filed a bug here: > > https://bugs.launchpad.net/neutron/+bug/1430984 > > > > I've outlined the path I'd like to take in the bug description. > > > > ----- Original Message ----- > >> +1 on avoiding changes that break rolling upgrade. > >> > >> Rolling upgrade has been working so far (at least from my > >> perspective), and as openstack adoption spreads, it will be > >> important for more and more users. > >> > >> How do we make rolling upgrade a "supported" part of Neutron? > > > > Finding a sane way to test it would be a start. I'm still > > looking... > > > >> > >> - Jack > >> > >>> -----Original Message----- From: Assaf Muller > >>> [mailto:amul...@redhat.com] Sent: Thursday, March 05, 2015 > >>> 11:59 AM To: OpenStack Development Mailing List (not for usage > >>> questions) Subject: Re: [openstack-dev] [Neutron] Issue when > >>> upgrading from Juno to Kilo due to agent report_state RPC > >>> namespace patch > >>> > >>> > >>> > >>> ----- Original Message ----- > >>>> To turn this stuff off, you don't need to revert. I'd > >>>> suggest just setting the namespace contants to None, and that > >>>> will result in the same thing. > >>>> > >>>> > >>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/common/constants.py# > >>> > >>> > n152 > >>>> > >>>> It's definitely a non-backwards compatible change. That was > >>>> a conscious choice as the interfaces are a bit of a tangled > >>>> mess, IMO. The non-backwards compatible changes were simpler > >>>> so I went that route, because as far as I could tell, rolling > >>>> upgrades were not supported. If they do work, it's due to > >>>> luck. There's multiple things including the lack of testing > >>>> this scenario to lack of data versioning that make it a > >>>> pretty shaky area. > >>>> > >>>> However, if it worked for some people, I totally get the > >>>> argument against breaking it intentionally. As mentioned > >>>> before, a quick fix if needed is to just set the namespace > >>>> constants to None. If someone wants to do something to make > >>>> it backwards compatible, that's even better. > >>>> > >>> > >>> I sent out an email to the operators list to get some > >>> feedback: > >>> http://lists.openstack.org/pipermail/openstack-operators/2015-March/006429.html > >>> > >>> > >>> > And at least one operator reported that he performed a rolling Neutron > >>> upgrade from I to J successfully. So, I'm agreeing with you > >>> agreeing with me that we probably don't want to mess this up > >>> knowingly, even though there is no testing to make sure that it > >>> keeps working. > >>> > >>> I'll follow up on IRC with you to figure out who's doing what. > >>> > >>>> -- Russell Bryant > >>>> > >>>> On 03/04/2015 11:50 AM, Salvatore Orlando wrote: > >>>>> To put in another way I think we might say that change > >>>>> 154670 broke backward compatibility on the RPC interface. > >>>>> To be fair this probably happened because RPC interfaces > >>>>> were organised in a way such that this kind of breakage was > >>>>> unavoidable. > >>>>> > >>>>> I think the strategy proposed by Assaf is a viable one. The > >>>>> point about being able to do rolling upgrades only from > >>>>> version N to N+1 is a sensible one, but it has more to do > >>>>> with general backward compability rules for RPC > >>>>> interfaces. > >>>>> > >>>>> In the meanwhile this is breaking a typical upgrade > >>>>> scenario. If a fix allowing agent state updates both > >>>>> namespaced and not is available today or tomorrow, that's > >>>>> fine. Otherwise I'd revert just to be safe. > >>>>> > >>>>> By the way, we were supposed to have already removed all > >>>>> server rpc callbacks in the appropriate package... did we > >>>>> forget out this one or is there a reason for which it's > >>>>> still in neutron.db? > >>>>> > >>>>> Salvatore > >>>>> > >>>>> On 4 March 2015 at 17:23, Miguel Ángel Ajo > >>>>> <majop...@redhat.com <mailto:majop...@redhat.com>> wrote: > >>>>> > >>>>> I agree with Assaf, this is an issue across updates, and we > >>>>> may want (if that’s technically possible) to provide access > >>>>> to those functions with/without namespace. > >>>>> > >>>>> Or otherwise think about reverting for now until we find a > >>>>> migration strategy > >>>>> > >>>>> > >>> https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:m > >>> > >>> > aster+topic:bp/rpc-docs-and-namespaces,n,z > >>>>> > >>>>> > >>>>> Best regards, Miguel Ángel Ajo > >>>>> > >>>>> On Wednesday, 4 de March de 2015 at 17:00, Assaf Muller > >>>>> wrote: > >>>>> > >>>>>> Hello everyone, > >>>>>> > >>>>>> I'd like to highlight an issue with: > >>>>>> https://review.openstack.org/#/c/154670/ > >>>>>> > >>>>>> According to my understanding, most deployments upgrade > >>>>>> the controllers first and compute/network nodes later. > >>>>>> During that time period, all agents will fail to report > >>>>>> state as they're sending the report_state message > >>>>>> outside of any namespace while the server is expecting > >>>>>> that message in a namespace. This is a show stopper as > >>>>>> the Neutron server will think all of its agents are > >>>>>> dead. > >>>>>> > >>>>>> I think the obvious solution is to modify the Neutron > >>>>>> server code so that it accepts the report_state method > >>>>>> both in and outside of the report_state RPC namespace and > >>>>>> chuck that code away in L (Assuming we support rolling > >>>>>> upgrades only from version N to N+1, which while is > >>>>>> unfortunate, is the behavior I've seen in multiple places > >>>>>> in the code). > >>>>>> > >>>>>> Finally, are there additional similar issues for other > >>>>>> RPC methods placed in a namespace this cycle? > >>>>>> > >>>>>> > >>>>>> Assaf Muller, Cloud Networking Engineer Red Hat > >>>>>> > >> > >> __________________________________________________________________________ > >> > >> > OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: > >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > >> > > __________________________________________________________________________ > > > > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQEcBAEBAgAGBQJVAMlQAAoJEC5aWaUY1u57PNQH/A6lwRfjplG8A5C9gW6Bmz4P > ArAbWfTeAfJWTo/oHH5gqy1Ni8i3Rkax3gWY9RtjaCYIy0jRQOHCaUY1AP5yFpLl > A2gqFq1ofq9U41sWDMxohbiVzfWrGN/PiiNMuoeXOtChO6EctToC0bpVthrfvG49 > H65Luhrkiva3Sg2yb+V1rN54hqQSQIKlCLalwUI7y2PQ+QexKxDl+fKsPl9NukLK > VqwHuelTJWCDvSCe+mEFsrkaaIbici/F5pxJV63slOq3wD5kpHirGVMyU/JWZRqB > rgO8zqFT1F/hH0NLmjtT1ojSVIIIgmAPo6vRejCjnVJGG3rYQHnKlh3wlIhfFiw= > =uTXa > -----END PGP SIGNATURE----- > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev