Flavio's fix was merged to master and back ported to the stable/kilo master. We're hoping to run it through our upgrade testing tomorrow.
On Tue, Jun 9, 2015 at 8:33 AM, Flavio Percoco <fla...@redhat.com> wrote: > On 09/06/15 07:38 +0000, Bhandaru, Malini K wrote: > >> A DB migrate script that puts some token default string only for glance >> properties that are NULL. Should not change anything else. >> Hope it works Flavio. >> Regards >> Malini >> > > Proposed fix, please read the commit message and comments: > > https://review.openstack.org/#/c/189661/ > > > >> -----Original Message----- >> From: Flavio Percoco [mailto:fla...@redhat.com] >> Sent: Tuesday, June 09, 2015 12:33 AM >> To: Bhandaru, Malini K >> Cc: OpenStack Development Mailing List (not for usage questions); >> openstack-operators@lists.openstack.org >> Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo upgrade >> & Nova >> >> On 09/06/15 09:22 +0200, Flavio Percoco wrote: >> >>> On 09/06/15 07:09 +0000, Bhandaru, Malini K wrote: >>> >>>> Flavio, would a DB script that writes an empty string or NOP or >>>> something instead of NULL In the column do the trick? >>>> Then the problem degenerates to a new DB upgrade script. >>>> >>> >>> I now remembered about a change[0] - that I wrote myself - that >>> required a bump on the API version - which we did - that allows None >>> values to be returned in the API. This is probably what is causing this >>> behavior. >>> >>> A DB migration would certainly work, I'd love to avoid it but I guess >>> that's the best solution in this case. >>> >> >> Actually, we can't just migrate the database as there might be custom >> properties that were explicitly set to None. >> >> I'll keep you posted >> >> >>> Cheers, >>> Flavio >>> >>> [0] https://review.openstack.org/#/c/138184/ >>> >>> >>>> Regards >>>> Malini >>>> >>>> -----Original Message----- >>>> From: Flavio Percoco [mailto:fla...@redhat.com] >>>> Sent: Monday, June 08, 2015 11:47 PM >>>> To: OpenStack Development Mailing List (not for usage questions) >>>> Cc: openstack-operators@lists.openstack.org >>>> Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo >>>> upgrade & Nova >>>> >>>> On 08/06/15 11:46 -0400, Clayton O'Neill wrote: >>>> >>>>> We tested testing Kilo upgrades in our hardware dev environments last >>>>> week and the second time through ran into this bug which right now is >>>>> probably a show-stopper for us. >>>>> >>>>> https://bugs.launchpad.net/glance/+bug/1419823 >>>>> >>>>> The issue here is that the v1 Glance API allows you to create images >>>>> with properties that are 'NULL' in the Glance database. For example: >>>>> >>>>> glance image-create --name cirros_test --disk-format qcow2 >>>>> --container-format bare --file cirros-0.3.4-x86_64-disk.img >>>>> --is-public True --is-protected True --progress --property >>>>> description= >>>>> >>>>> It's apparently also fairly easy to end up with a NULL description >>>>> when editing images properties via Horizon. >>>>> >>>>> The issue is that the v2 Glance API returns these NULL properties to >>>>> the client, which then validates them against the schema returned by >>>>> the v2 API. >>>>> This schema returns specifies that the description property *must* be >>>>> a string. >>>>> >>>>> In the Kilo release, Nova has been changed to use the v2 API, so >>>>> suddenly this matters. The net effect is that end users can pretty >>>>> easily create properties with NULL values, and then won't be able to >>>>> boot instances using those images. >>>>> What makes this worse is that it's completely opaque to end users, >>>>> since this just reports that no node was available to schedule the >>>>> instance. >>>>> >>>>> However, Nova *only* uses the v2 api to list images if the >>>>> glance.allowed_direct_url_schemes config key is set in the config file. >>>>> However, this config item defaults to an empty array, meaning that by >>>>> default it's *always* set. There doesn't appear to be a way to unset >>>>> a value with oslo-config that has a default value, blocking off that >>>>> route to work around the issue. Disabling the v2 Glance API we don't >>>>> think will work, since Nova appears to assume the v2 API is available. >>>>> >>>> >>>> AFAIK, Nova supports V2 image-lists since before Juno when the >>>> allowed_direct_url_schemes config option is set. Are you referring to >>>> another change? Has the default been changed in Nova? I'm asking because I >>>> was working on this migration to V2 and we decided to postpone it to L. >>>> >>>> >>>>> Another work around we've looked at is to change the DB schema for >>>>> image properties (yuck) to not allow NULL values. This results in >>>>> Glance returning a >>>>> 500 error since glance-api is attempting to insert an invalid value. >>>>> This is better than instances failing in an opaque fashion, but still >>>>> pretty horrible. >>>>> >>>>> Has anyone else run into this issue yet? Are there other work >>>>> arounds that we've not thought of other than "Don't create images with >>>>> NULL properties?" >>>>> User education is definitely an option, but given the failure mode, >>>>> it's not a great solution for us. >>>>> >>>> >>>> I believe this needs to be fixed in the client rather than the API >>>> and/or the schemas. I'll take a look at this right away. >>>> >>>> Another workaround could be updateting `schema-image.json` to add the >>>> schemas that are missing and let the client download the final schema from >>>> the V2 API. >>>> >>>> Keep an eye on the bug, patches coming your way (assuming what I have >>>> in mind will work). >>>> Flavio >>>> >>>> -- >>>> @flaper87 >>>> Flavio Percoco >>>> >>> >>> -- >>> @flaper87 >>> Flavio Percoco >>> >> >> >> >> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> -- >> @flaper87 >> Flavio Percoco >> > > -- > @flaper87 > Flavio Percoco > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
