Matt, We discussed this at the Trove meeting. Here's my current understanding of the situation:
1. Your change https://review.openstack.org/#/c/290048/ which reverts the trove client side of the change should be merged. 2. This change (the Trove API side) https://review.openstack.org/#/c/245845/ should also be reverted. I'm assuming that you'll submit a change for this as well for completeness. 3. We need to blacklist python-troveclient 2.1.0 on Liberty, this is your change https://review.openstack.org/#/c/290021/ 4. We need to blacklist python-troveclient 2.1.0 on master, this is currently *NOT* what your change https://review.openstack.org/290168 does. I disagree with the approach of just increasing the minimum requirement from 1.2.0 to >=2.1.0. 5. We need to request a new python-troveclient. Whether it should be called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of an abundance of caution, I'm going to look to #openstack-release/#openstack-infra for guidance on this. Thanks, -amrith > -----Original Message----- > From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com] > Sent: Wednesday, March 09, 2016 12:42 PM > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly > failing in stable branches; bug 1538506 > > > > On 3/9/2016 8:54 AM, Amrith Kumar wrote: > > Matt, > > > > As I understand it, you have 4 changes up for review. > > > > Change [1] seeks to revert the change [6] to python-troveclient on > > master and reinstate the slave_of parameter. > > > > Change [2] is doing for stable/liberty, the same things that were > > done for master in [9] on master, and [7] on Kilo earlier. > > > > Change [3] is looking to blacklist python-troveclient 2.1.0 on > > stable/liberty. > > > > Change [4] is looking to bump the minimum python-troveclient > version > > on master to 2.1.0. > > Right, and there was actually another for that beat me to this: > > https://review.openstack.org/#/c/290115/ > > > > > Here are three questions I have: > > > > (1) Wouldn't backward compatibility also require that you revert the > > other change that removed slave_of on the server [5] ? After all, just > > like a python client that calls create() with slave_of, a REST client > > could call the endpoint with slave_of. What is the policy for REST API > > backward compatibility? > > I guess it depends on the Trove team. In Nova, backward compatibility in > the API is serious business and that's why we have left all sorts of warts > in the v2.0 API and couldn't remove them. But with the v2.1 API and > microversions, we're able to move the API forward (Ironic also has > microversions, and I think cinder/neutron/keystone are working on adding > that support). > > So if maintaining backward compatibility in the trove API is important to > the trove team and it's users, then yes the API change should probably be > reverted. For anyone doing CD with Trove, they are already broken, but > people upgrading from liberty to mitaka could save themselves the break. > > > > > (2) Wouldn't [4] just block 2.1.0 in global-requirements on master > > (mitaka)? > > I'm not sure I understand, [4] here bumps the minimum required version of > python-troveclient to be 2.1.0, not block it. As noted in the review, I > don't know that it's really necessary to bump to 2.1.0 iff we land and > release [1]. > > > > > (3) Something, potentially your patch set [2], should also fix the > > test that is invoking create() with slave_of, no? > > [2] is stable/liberty which still has slave_of in the create API, so I > don't think that needs to be fixed in stable/liberty. > > > > > -amrith > > > > [1] https://review.openstack.org/290048/ > > [2] https://review.openstack.org/290023/ > > [3] https://review.openstack.org/290021/ > > [4] https://review.openstack.org/290168/ > > [5] https://review.openstack.org/#/c/245845/ > > [6] https://review.openstack.org/#/c/245738/ > > [7] https://review.openstack.org/#/c/246735/ > > > > On 03/08/2016 08:24 PM, Matt Riedemann wrote: > >> > >> > >> On 3/8/2016 4:44 PM, Matt Riedemann wrote: > >>> > >>> > >>> On 3/8/2016 3:17 PM, Matt Riedemann wrote: > >>>> > >>>> > >>>> On 3/8/2016 1:18 PM, Amrith Kumar wrote: > >>>>> Matt, > >>>>> > >>>>> See inline below. > >>>>> > >>>>> -amrith > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com] > >>>>>> Sent: Tuesday, March 08, 2016 2:00 PM > >>>>>> To: openstack-dev@lists.openstack.org > >>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests > >>>>>> randomly failing in stable branches; bug 1538506 > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 3/8/2016 12:52 PM, Matt Riedemann wrote: > >>>>>>> > >>>>>>> > >>>>>>> On 3/8/2016 12:35 PM, Amrith Kumar wrote: > >>>>>>>> Matt, > >>>>>>>> > >>>>>>>> The correct solution for liberty is that we should fix the tests. > >>>>>>>> Here's why I believe that this is the case. > >>>>>>>> > >>>>>>>> In pertinent part, the backtrace from your bug includes: > >>>>>>>> > >>>>>>>> 2016-01-27 07:02:06.777 | File > >>>>>>>> "/home/jenkins/workspace/periodic-trove-python27-liberty/trove/ > >>>>>>>> tests/ > >>>>>>>> > >>>>>>>> api/replication.py", > >>>>>>>> line 83, in create_slave > >>>>>>>> 2016-01-27 07:02:06.777 | slave_of=instance_info.id) > >>>>>>>> > >>>>>>>> This is in the tests, not the client. > >>>>>>>> > >>>>>>>> The test is now generating a parameter that the client no > >>>>>>>> longer understands. > >>>>>>>> > >>>>>>>> For a user, here are the various situations that could occur. > >>>>>>>> > >>>>>>>> 1. User running python-troveclient from the latest 2.1.0 and a > >>>>>>>> server from liberty. All is good. > >>>>>>>> 2. User running an old python-troveclient and a server from > >>>>>>>> liberty. > >>>>>>>> All is good. > >>>>>>> > >>>>>>> Will this work with python-troveclient 1.2.0 which is the > >>>>>>> minimum required troveclient for stable/liberty? > >>>>>>> > >>>>>>> https://github.com/openstack/requirements/blob/stable/liberty/gl > >>>>>>> obal-r > >>>>>>> > >>>>>>> equirements.txt#L174 > >>>>>>> > >>>>>>> > >>>>>>>> 3. User running an old python-troveclient and a new server from > >>>>>>>> mitaka, this is not supported. > >>>>>>> > >>>>>>> How is this not supported? If it's not supported, the minimum > >>>>>>> required version of troveclient in master g-r is wrong, since it > >>>>>>> hasn't changed since liberty: > >>>>>>> > >>>>>>> https://github.com/openstack/requirements/blob/master/global-req > >>>>>>> uireme > >>>>>>> > >>>>>>> nts.txt#L202 > >>>>>> > >>>>>> I've confirmed that running master (mitaka) unit tests for trove > >>>>>> against python-troveclient 1.2.0 don't work: > >>>>>> > >>>>>> http://paste.openstack.org/show/489719/ > >>>>> > >>>>> [amrith] Just like the bug you listed, this appears to be a case > >>>>> of a broken test. Would you please LP it. > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> 4. User running a current python-troveclient from the latest > >>>>>>>> 2.1.0 and a server from mitaka, All is good. > >>>>>>>> > >>>>>>>> What you have here is a case where a test is generating an > >>>>>>>> argument > >>>>>>>> (slave_of) that the client (master, 2.1.0) no longer understands. > >>>>>>>> There's nothing amiss there, except that the test needs to be > >>>>>>>> fixed. > >>>>>>>> > >>>>>>>> When you get back, let's continue the conversation either in > >>>>>>>> email or IRC #openstack-dev. Looking forward to hearing your > >>>>>>>> feedback on this. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> > >>>>>>>> -amrith > >>>>>>>> > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com] > >>>>>>>>> Sent: Tuesday, March 08, 2016 12:11 PM > >>>>>>>>> To: openstack-dev@lists.openstack.org > >>>>>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests > >>>>>>>>> randomly failing in stable branches; bug 1538506 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 3/8/2016 10:17 AM, Matt Riedemann wrote: > >>>>>>>>>> This is a call for help on resolving bug 1538506 [1] where > >>>>>>>>>> the proboscis tests randomly fail on the stable branches with > >>>>>>>>>> something > >>>>>>>>> like: > >>>>>>>>>> > >>>>>>>>>> TypeError: create() got an unexpected keyword argument > 'slave_of' > >>>>>>>>>> > >>>>>>>>>> Craig Vyvial has a proposed stable/kilo change [2] but it has > >>>>>>>>>> some issues, at least from me as a reviewer of that change. > >>>>>>>>>> > >>>>>>>>>> The tests are failing the periodic-stable job daily [3]. > >>>>>>>>>> > >>>>>>>>>> Can we get someone to help out with this? Otherwise I'm > >>>>>>>>>> inclined to say the tests should be disabled on the stable > >>>>>>>>>> branches, but that's pretty drastic. > >>>>>>>>>> > >>>>>>>>>> Note that this is the kind of thing that breaks stable branch > >>>>>>>>>> policy, which is going to be part of a review when applying > >>>>>>>>>> for the stable:follows-policy tag. [4] And the > >>>>>>>>>> stable:follows-policy tag may be used to determine when a > >>>>>>>>>> stable branch for a project goes end of life if it's not > >>>>>>>>>> being maintained. > >>>>>>>>>> > >>>>>>>>>> [1] https://bugs.launchpad.net/trove/+bug/1538506 > >>>>>>>>>> [2] https://review.openstack.org/#/c/276934/ > >>>>>>>>>> [3] http://goo.gl/fqf11U > >>>>>>>>>> [4] > >>>>>>>>>> http://governance.openstack.org/reference/tags/stable_follows > >>>>>>>>>> -polic > >>>>>>>>>> > >>>>>>>>>> y.h > >>>>>>>>>> tml > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> This is my solution for liberty, cap python-troveclient<2.1.0 > >>>>>>>>> in > >>>>>>>>> global- requirements on stable/liberty [1]. > >>>>>>>>> > >>>>>>>>> Then there is a change to trove on stable/liberty to use the > >>>>>>>>> g-r version range for python-troveclient for unit tests [2]. > >>>>>>>>> > >>>>>>>>> Alternatively, we could avoid the cap in stable/liberty by: > >>>>>>>>> > >>>>>>>>> 1. Reverting https://review.openstack.org/#/c/245738/ > >>>>>>>>> 2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release > >>>>>>>>> python- troveclient 2.2.0. > >>>>>>>>> > >>>>>>>>> It's getting late in the mitaka release to be messing with > >>>>>>>>> this though since we're already past client freeze. > >>>>>>>>> > >>>>>>>>> [1] https://review.openstack.org/#/c/290021/ > >>>>>>>>> [2] https://review.openstack.org/#/c/290023/ > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Matt Riedemann > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ______________________________________________________________ > >>>>>>>>> ______ > >>>>>>>>> > >>>>>>>>> ______ > >>>>>>>>> > >>>>>>>>> OpenStack Development Mailing List (not for usage questions) > >>>>>>>>> Unsubscribe: > >>>>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack- > >>>>>>>>> dev > >>>>>>>> > >>>>>>>> _______________________________________________________________ > >>>>>>>> ______ > >>>>>>>> > >>>>>>>> _____ > >>>>>>>> > >>>>>>>> OpenStack Development Mailing List (not for usage questions) > >>>>>>>> Unsubscribe: > >>>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d > >>>>>>>> ev > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Matt Riedemann > >>>>>> > >>>>>> > >>>>>> _________________________________________________________________ > >>>>>> _________ > >>>>>> > >>>>>> > >>>>>> > >>>>>> OpenStack Development Mailing List (not for usage questions) > >>>>>> Unsubscribe: > >>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>> > >>>>> __________________________________________________________________ > >>>>> ________ > >>>>> > >>>>> > >>>>> > >>>>> OpenStack Development Mailing List (not for usage questions) > >>>>> Unsubscribe: > >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>> > >>>> > >>>> Well, the point I'm trying to get to is, I don't think troveclient > >>>> < > >>>> 2.1.0 will work with the trove-api in mitaka. > >>>> > >>>> For example, in my change to revert the slave_of removal in the > client: > >>>> > >>>> https://review.openstack.org/#/c/290048/ > >>>> > >>>> That fails in the API: > >>>> > >>>> http://logs.openstack.org/48/290048/2/check/gate-trove-functional-d > >>>> svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_79 > >>>> 0 > >>>> > >>>> > >>>> > >>>> > >>>> "Validation error: instance Additional properties are not allowed > >>>> (u'slave_of' was unexpected)" > >>>> > >>>> So if my client application that I wrote in liberty was using > >>>> slave_of, it's happily working until the cloud that my application > >>>> is running on upgrades to mitaka, then things break. > >>>> > >>>> Part of the issue here is the backward incompatible change in the > >>>> trove API. The other part is, the trove API in mitaka requires > >>>> troveclient>=2.1.0 because that's what removed slave_of. > >>>> > >>>> Right? > >>>> > >>>> I'm trying to sort out what a valid troveclient is for master, > >>>> because I don't really trust what's in global-requirements since > >>>> trove hasn't been testing with those versions, at least not at the > >>>> lower bound of 1.2.0. > >>>> > >>> > >>> Here is a change that bumps the minimum required version of > >>> python-troveclient for mitaka to 2.1.0: > >>> > >>> https://review.openstack.org/#/c/290168/ > >>> > >> > >> Here is my preferred option now. > >> > >> 1. Revert the slave_of removal from python-troveclient and provide it > >> as a compat shim until liberty-eol: > >> > >> https://review.openstack.org/#/c/290048/ > >> > >> If slave_of is used, a deprecation warning is emitted (which we > >> should have been doing when this was deprecated in the first place). > >> It also never sends slave_of to the API, it's just a proxy to > >> replica_of. So this should work with both the liberty and mitaka > >> versions of the trove API. > >> > >> 2. Block python-troveclient 2.1.0 from stable/liberty g-r: > >> > >> https://review.openstack.org/#/c/290021/ > >> > >> That blocks the bad 2.1.0 version from being used in stable/liberty > >> which is needed for... > >> > >> 3. Using the g-r versions of python-troveclient when testing on > >> stable/liberty: > >> > >> https://review.openstack.org/#/c/290023/ > >> > >> If trove is following global-requirements, it should be using the > >> versions of python-troveclient from global-requirements rather than > >> installing it from the latest tarball on trunk. master and > >> stable/kilo are already doing this. > >> > > > > > > > > > > ______________________________________________________________________ > > ____ OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > > Thanks, > > Matt Riedemann > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev