Re: [Openstack-operators] [Large deployments] Openstack Large deployment using DVR

[Clayton O'Neill]

On Tue, Nov 1, 2016 at 7:19 AM, Satyanarayana Patibandla
 wrote:
> We are planning for a new Openstack large deployment ( Around 25 K VMs). For
> external routing we want to use DVR for instances with floating IP and
> Network node L3 agent for instances with private IP. Our deployment is DVR
> with L3 HA.It seems still many people not yet started using DVR in
> production for large deployments. Did anyone use DVR in their Large/Medium
> production deployment. If you can share your experiences that will be
> helpful to us and that will save most of our time.

I've asked this question in operator sessions at the Summits and at
the NYC operators mid-cycle with not much response.  In the Neutron
feedback session last week, there was one person that said he was
running DVR, but that it had architectural issues with IPv6.  I got
the impression that his environment was much smaller than what you're
considering.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Liberty and OVS Agent restarts

[Clayton O'Neill]

I’ve tried it with both a blank value and the specific value.  It
doesn’t appear to make a difference.

In other news, Assaf Muller has upgraded the priority of the bug from
low to high.

On Fri, Feb 12, 2016 at 9:27 AM, Matt Kassawara <mkassaw...@gmail.com> wrote:
> Out of curiosity, what do you have for the "external_network_bridge" option
> in the L3 agent config?
>
> On Wed, Feb 10, 2016 at 2:42 PM, Bajin, Joseph <jba...@verisign.com> wrote:
>>
>> Clayton,
>>
>> This is really good information.
>>
>> I’m wondering how we can help support you and get the necessary dev
>> support to get this resolved sooner than later. I totally agree with you
>> that this should be backported to at least Liberty.
>>
>> Please let me know how I and other can help!
>>
>> —Joe
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 2/10/16, 8:55 AM, "Clayton O'Neill" <clay...@oneill.net> wrote:
>>
>> >Summary: Liberty OVS agent restarts are better, but still need work.
>> >See: https://bugs.launchpad.net/neutron/+bug/1514056
>> >
>> >As many of you know, Liberty has a fix for OVS agent restarts such
>> >that it doesn’t dump all flows when starting, resulting in a loss of
>> >traffic.  Unfortunately, Liberty neutron still has issues with OVS
>> >agent restarts.  The fix that went into Liberty prevents it from
>> >dropping flows on the br-tun and br-int bridges and that helps
>> >greatly, but the br-ex bridge still has it’s flows cleared on startup.
>> >
>> >You may be thinking: Wait, br-ex only has like 3 flows on it, how can
>> >that be a problem?  The issue appears to be that the br-ex flows are
>> >cleared early and not setup again until late in the process.  This
>> >means that routers on the node where OVS agent is lose network
>> >connectivity for the majority of the restart time.
>> >
>> >I did some testing with this yesterday, comparing a few scenarios with
>> >100 FIPS, 100 instances and various scenarios for routers.  You can
>> >find the the complete data here:
>>
>> > >https://docs.google.com/spreadsheets/d/1ZGra_MszBlL0fNsFqd4nOvh1PsgWu58-GxEeh1m1BPw/edit?usp=sharing
>> >
>> >The summary looks like this:
>> >100 routers, 100 networks, 100 floating ips, 100 instances, single node
>> > test:
>> >Kilo average outage time: 47 seconds
>> >Liberty average outage time: 37 seconds
>> >
>> >1 router, 1 network, 100 floating ips, 100 instances, single node test:
>> >Kilo average outage time: 46 seconds
>> >Liberty average outage time: 13 seconds
>> >
>> >1 router, 1 network, 100 floating its, 100 instances, router on a
>> >separate node, all instances on a single node, OVS restart on compute
>> >node:
>> >Kilo average outage time: 25 seconds
>> >Liberty average outage time: 0 to 1 seconds
>> >
>> >I did my testing using 1 second pings using fping to all of the
>> >floating IPs.  With the last test, it frequently lost no packets, and
>> >as a result I was not really able to test the scenario other than to
>> >qualify it as good.
>> >
>> >This is a huge operational issue for us and I suspect for many of the
>> >rest of you using OVS.  I’d encourage everyone that is using OVS to
>> >register interest in having this fixed in the LP bug
>> >(https://bugs.launchpad.net/neutron/+bug/1514056).  Right now this bug
>> >as marked as low priority.
>> >
>> >___
>> >OpenStack-operators mailing list
>> >OpenStack-operators@lists.openstack.org
>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Liberty and OVS Agent restarts

[Clayton O'Neill]

Summary: Liberty OVS agent restarts are better, but still need work.
See: https://bugs.launchpad.net/neutron/+bug/1514056

As many of you know, Liberty has a fix for OVS agent restarts such
that it doesn’t dump all flows when starting, resulting in a loss of
traffic.  Unfortunately, Liberty neutron still has issues with OVS
agent restarts.  The fix that went into Liberty prevents it from
dropping flows on the br-tun and br-int bridges and that helps
greatly, but the br-ex bridge still has it’s flows cleared on startup.

You may be thinking: Wait, br-ex only has like 3 flows on it, how can
that be a problem?  The issue appears to be that the br-ex flows are
cleared early and not setup again until late in the process.  This
means that routers on the node where OVS agent is lose network
connectivity for the majority of the restart time.

I did some testing with this yesterday, comparing a few scenarios with
100 FIPS, 100 instances and various scenarios for routers.  You can
find the the complete data here:
https://docs.google.com/spreadsheets/d/1ZGra_MszBlL0fNsFqd4nOvh1PsgWu58-GxEeh1m1BPw/edit?usp=sharing

The summary looks like this:
100 routers, 100 networks, 100 floating ips, 100 instances, single node test:
Kilo average outage time: 47 seconds
Liberty average outage time: 37 seconds

1 router, 1 network, 100 floating ips, 100 instances, single node test:
Kilo average outage time: 46 seconds
Liberty average outage time: 13 seconds

1 router, 1 network, 100 floating its, 100 instances, router on a
separate node, all instances on a single node, OVS restart on compute
node:
Kilo average outage time: 25 seconds
Liberty average outage time: 0 to 1 seconds

I did my testing using 1 second pings using fping to all of the
floating IPs.  With the last test, it frequently lost no packets, and
as a result I was not really able to test the scenario other than to
qualify it as good.

This is a huge operational issue for us and I suspect for many of the
rest of you using OVS.  I’d encourage everyone that is using OVS to
register interest in having this fixed in the LP bug
(https://bugs.launchpad.net/neutron/+bug/1514056).  Right now this bug
as marked as low priority.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] setting dhcp_domain per project

[Clayton O'Neill]

I believe this is coming in Liberty,  based on this talk:
https://www.openstack.org/summit/tokyo-2015/videos/presentation/get-your-instance-by-name-integration-of-nova-neutron-and-designate

On Thu, Nov 19, 2015 at 7:10 AM, Davíð Örn Jóhannsson 
wrote:

> I've been looking into changing the search domain provided by the dhcp
> service that is propagated to /etc/resolv.conf via dhcp in openstack.
>
>
> The documentation points to changing dhcp_domain in dhcp_agent.ini but
> would that not result in all projects and all dhcp agents in openstack
> sending out the same search domain.
>
>
> I need to be able to configure for example
>
> example.com
>
> dev.example.com
>
>
> in different projects, is this not possible for example on a per dhcp per
> network in horizon or something like that?
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OPs Midcycle location discussion.

[Clayton O'Neill]

On Wed, Nov 18, 2015 at 4:24 PM, Erik McCormick 
wrote:

> > 2) More technically we'll need to address the challenge of local
> > sound, how to we ensure all the mostly spontaious talk in a large work
> > session makes it to remote participants.  Passing a mic is a bit
> > cumbersome and hard to enforce, while mic'ing the room to properly get
> > ambient sound isn't likely something we can do without significant
> > professional help.
> >
>
> This is a technical issue that needs dealing with for sure. Passing a
> mic is absolutely not practical, so it would need to be some sort of
> ambient thing. We can't even get people to consistently go to a mic
> during Q at the end of summit presentations. I've seen enough
> teleconferencing systems that cover large boardroom settings to know
> that such things exist, but I have no specific knowledge of what they
> are, if they can be rented, or how much they cost. That will require a
> lot of digging into.


A key difference between the Ceph summit and the Ops Mid-cycle appears to
be that the Ops Mid-cycle is at least an order of magnitude bigger.

For a good amount of the Philly and Palo Alto mid-cycles there were 200ish
people in a single room.  Audio was a problem even for people sitting in
the room at times.   I think this is beyond “large boardroom” size.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-operators][osops][tools-generic] Coding Standards/Coding Linters for tools-generic

[Clayton O'Neill]

On Tue, Nov 17, 2015 at 12:56 PM, JJ Asghar  wrote:

> Yep, that's the can o'worms I was talking about. I was thinking that we
> could just steal this .rubocop.yml[1], as it's a "sane" default for us
> at Chef. If you take a look at it you'll see its pretty simple which is
> what we're looking for.
>
> Speaking of this .rubocop.yml I'll add it to the etherpad.
>

Sounds good to me.  I would just start with that and not change any of the
preferences unless there is overwhelming support for doing so.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-operators][osops][tools-generic] Coding Standards/Coding Linters for tools-generic

[Clayton O'Neill]

I think it’s a good idea.  I think scripts and such that don’t pass the
listing tools can go into the contrib repos and if they get cleaned up then
they can move over to the regular ones.

I don’t actually like some of the PEP8 and bashate rules, but I’d rather
have a consistent style than have them in everyone’s personal style.  I
find less opinionated tools like rubocop to be less useful, since you end
up bogged down in arguing about which style options to choose.

On Mon, Nov 16, 2015 at 7:03 PM, JJ Asghar  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> NOTE: I get what I'm about to propose will open a HUGE can of worms, but
> we need it, so I'll start the conversation.
>
> We had some initial discussion and thoughts on coding standards when we
> first started this project. It got shot down, but not beforeChristian
> Berendt came up with review[1].
>
> If you don't know, bashate[2] is a pep8 coding standard for bash. I
> think it's a great starting point for our tools-generic repo.
>
> Are there any objections or concerns with this? If there isn't any,
> voiced in a timely fashion, I'll take the action item of getting the
> review re-enabled.
>
> To the can o'worms: I think we should probably set up linters for any
> python, ruby, bash, etc scripts also. It's hard to cargo cult these from
> other projects in the OpenStack ecosystem; which I understand can be
> different per different project. This can be a religious debate, and I
> want to acknowledge that people feel strongly about this; but we all
> need to come together to get this project off the ground. I'd like to go
> as _simple_ as possible, then we can add more opinionated styles, if
> desired, as our project grows.
>
> I propose people post different standards here[3] and when we have our
> next meeting we vote to figure out what we should have. If you can't
> attend the meeting please put +1s by what you'd like and we can take
> that in for consideration.
>
> Please if you have any concerns, questions, thoughts don't hesitate to
> reach out.
>
>
> [1]: https://review.openstack.org/#/c/229029
> [2]: https://github.com/openstack-dev/bashate
> [3]: https://etherpad.openstack.org/p/osops-coding-standards
>
>
> - --
> Best Regards,
> JJ Asghar
> c: 512.619.0722 t: @jjasghar irc: j^2
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2
> Comment: GPGTools - https://gpgtools.org
>
> iQIcBAEBCgAGBQJWSm7rAAoJEDZbxzMH0+jTJDYQAJ4dNZQhkH3jmLLBo7iqNXYN
> gUngDUHfRL0UY63dC/3v4V7OtH+Eg5QNPXSYvw1CzLZm+KfvfaYTb4PIY9ZAt83f
> 7h6dCgtF5XESHFHVNoEOAQYpTRM+99H0N0R5DE1alGow+5f2Rj9dWpBD59RYQbFR
> TX8IPvEPQRtAvfErt0/O6TxOLR64CSehYRyki3CWgVtBd1Q+cOl+0NV4+cneIqUk
> EBEzQglCxtRAb0yWcMVRp5kHzLOB2o+6xzb8PXKYyKr9J1j1e0W679A6qzE3Ut+w
> l69mXQxfz8x/0B9lW0xqipgYN+17mxdAnnu3Rhwr85HAMeI5O+M9w0CF7rxrwWII
> qZVb8/zZKfX3jK6hga6+mQZgZUAvmxGxV498TEkOZLIuqo+catH7SxADmwpafSuP
> J+TrAT97gZrKaGsBCS8Iwzs/4eBJUrL3enX0PXR+UcsCXAo5TjYYx34jwtqviNkI
> GNahmLuUoOsszg1WtNKFSs6Cc86u0GPoUer3VmUEwaVkxwyoIpYC5ePmaZZ6MFZb
> se75w5t0BUzO4StiJPV7C/U40SuA3DFG308DK1of/isOXJ6kYcraZOMutM+4xznZ
> UfCxVF65Tsq6+YRc6XCS5qBoEHGJkm/8iQiy88LlI5RYUcdBFnEBgxggnJWrWPIK
> Lsg7ynN9YPQ/0jaWfKxQ
> =RLkF
> -END PGP SIGNATURE-
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] OVS flow dump fix

[Clayton O'Neill]

As many of you know, Liberty includes a bug fix for OVS L2 agent dropping
all flows when it starts up.  Since this is a bug, I’d like to see this
back ported to Kilo if possible.

We’re planning to do an upgrade to the latest stable/kilo because of the
Nova NUMA issue anyway and we’re still using packages for Nova & Neutron.
If this can be back ported then our upgrade to Liberty would go a lot
smoother since this would be once less network related issue to deal with
during the upgrade.

I’d appreciate if anyone that feels the same could leave a comment to that
effect on the bug:

https://bugs.launchpad.net/neutron/+bug/1383674
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Adding v1 LIKE support to python-glanceclient releases 1.x.x

[Clayton O'Neill]

I ran into this issue on my laptop where I had just pip installed glance
client and gotten the latest version.  The script I was working on I want
to run in multiple environments, so it was easier to rely on the openstack
client which has the same CLI API (and generally easier to integrate with)
than to make it work with two versions of the glance client or to force a
specific version on systems I may not control.

I don't remember the exact behavior change that bit me, but since it was
trying to upload an image if it didn't already exist by name, I suspect
that it was the --is-public/--is-protected change.

I think one of the issues here is that a lot of our users also install from
pip, and from their standpoint this is an "unannounced" change.  Not
because it's literally unannounced, but because many of them probably don't
install specific versions of the clients even though we do recommend
specific versions.  From their standpoint I think they consider this
"unannounced" mostly because I doubt any of them follow any of the mailing
lists where it would have been announced.  I'm not sure what the right
answer here is, but it's definitely a difficult situation.

On Mon, Sep 14, 2015 at 7:18 AM, Flavio Percoco <fla...@redhat.com> wrote:

> On 09/09/15 21:46 -0400, Clayton O'Neill wrote:
>
>> I'd be glad to see the backwards compatibility parts go in.  I got bitten
>> by
>> this earlier this week and ended up switching my scripts over to using the
>> python-openstackclient library to work around it.
>>
>
>
> Hey Clayton,
>
> Thanks for the feedback.
>
> Could you be more precise on what incompatibility affected you?
>
> The patch that Nikhil linked in his email brings in several
> "compatibilities" with v1. I personally think they should be examined
> 1 by 1 rather than pulling them all in, hence my question.
>
> Switching to python-openstack client must have required some effort
> and I'm curious to know why you decided to do that rather than
> adapting your scripts to use the v2 cli. Do you have Glance's v2
> deployed?
>
> Ideally, I think we should just move to use openstackclient, really.
> But glanceclient is what we have now and that's what the Glance team
> has focused on the most lately so I'd appreaciate as much feedback as
> possible from you and others.
>
> Thanks for your time,
> Flavio
>
>
>
>> On Wed, Sep 9, 2015 at 6:41 PM, Nikhil Komawar <nik.koma...@gmail.com>
>> wrote:
>>
>>Hi all,
>>
>>We recently release python-glanceclient 1.0.0 and it has the default
>>shell version as v2. This may result into some scripts not detecting
>> the
>>change by default and discomfort to an extent.
>>
>>So, I am reaching out to this list with the hope of getting some
>>feedback on the requirements, pros and cons you all think exist for
>>adding some support for v1 like calls as hidden command to the default
>>python-glanceclient shell API that is v2 centric by default. This
>> should
>>unbreak the scripts to an extent and give a warning to users to update
>>the scripts in a stipulated time period so that they use the v2 API.
>>
>>Here's the proposed patch https://review.openstack.org/#/c/219802/ .
>> We
>>are not yet sure if we need to get it merged by tomorrow so that it can
>>be in stable/liberty by the end of the week. There has been one request
>>to get those in and the feedback we received from the developer
>>community was neutral.
>>
>>In order to form an opinion on what's best for our users, we need some
>>feedback on this topic. Please send us your thoughts as soon as
>> possible
>>and we will try to accommodate the same if permissible within the
>>technical limitations:
>>
>>1. Whether you would like these commands added as hidden commands so
>>that shell API works like before (to extent possible).
>>2. You would like to use v2 shell API of the client by default and
>> don't
>>care about this commit.
>>3. You don't care about the change. Your scripts are awesome and can
>>adjust to the upgrade of the client easily.
>>4. Anything else.
>>  --
>>
>>Thanks,
>>Nikhil
>>
>>
>>___
>>OpenStack-operators mailing list
>>OpenStack-operators@lists.openstack.org
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>>
> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> --
> @flaper87
> Flavio Percoco
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Adding v1 LIKE support to python-glanceclient releases 1.x.x

[Clayton O'Neill]

I'd be glad to see the backwards compatibility parts go in.  I got bitten
by this earlier this week and ended up switching my scripts over to using
the python-openstackclient library to work around it.

On Wed, Sep 9, 2015 at 6:41 PM, Nikhil Komawar 
wrote:

> Hi all,
>
> We recently release python-glanceclient 1.0.0 and it has the default
> shell version as v2. This may result into some scripts not detecting the
> change by default and discomfort to an extent.
>
> So, I am reaching out to this list with the hope of getting some
> feedback on the requirements, pros and cons you all think exist for
> adding some support for v1 like calls as hidden command to the default
> python-glanceclient shell API that is v2 centric by default. This should
> unbreak the scripts to an extent and give a warning to users to update
> the scripts in a stipulated time period so that they use the v2 API.
>
> Here's the proposed patch https://review.openstack.org/#/c/219802/ . We
> are not yet sure if we need to get it merged by tomorrow so that it can
> be in stable/liberty by the end of the week. There has been one request
> to get those in and the feedback we received from the developer
> community was neutral.
>
> In order to form an opinion on what's best for our users, we need some
> feedback on this topic. Please send us your thoughts as soon as possible
> and we will try to accommodate the same if permissible within the
> technical limitations:
>
> 1. Whether you would like these commands added as hidden commands so
> that shell API works like before (to extent possible).
> 2. You would like to use v2 shell API of the client by default and don't
> care about this commit.
> 3. You don't care about the change. Your scripts are awesome and can
> adjust to the upgrade of the client easily.
> 4. Anything else.
>
> --
>
> Thanks,
> Nikhil
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [glance] [nova] Glance bug with Kilo upgrade Nova

[Clayton O'Neill]

Flavio's fix was merged to master and back ported to the stable/kilo
master.  We're hoping to run it through our upgrade testing tomorrow.

On Tue, Jun 9, 2015 at 8:33 AM, Flavio Percoco fla...@redhat.com wrote:

 On 09/06/15 07:38 +, Bhandaru, Malini K wrote:

 A DB migrate script that puts some token default string only for glance
 properties that are NULL. Should not change anything else.
 Hope it works Flavio.
 Regards
 Malini


 Proposed fix, please read the commit message and comments:

 https://review.openstack.org/#/c/189661/



 -Original Message-
 From: Flavio Percoco [mailto:fla...@redhat.com]
 Sent: Tuesday, June 09, 2015 12:33 AM
 To: Bhandaru, Malini K
 Cc: OpenStack Development Mailing List (not for usage questions);
 openstack-operators@lists.openstack.org
 Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo upgrade
  Nova

 On 09/06/15 09:22 +0200, Flavio Percoco wrote:

 On 09/06/15 07:09 +, Bhandaru, Malini K wrote:

 Flavio, would a DB script that writes an empty string or NOP or
 something instead of NULL In the column do the trick?
 Then the problem degenerates to a new DB upgrade script.


 I now remembered about a change[0] - that I wrote myself - that
 required a bump on the API version - which we did - that allows None
 values to be returned in the API. This is probably what is causing this
 behavior.

 A DB migration would certainly work, I'd love to avoid it but I guess
 that's the best solution in this case.


 Actually, we can't just migrate the database as there might be custom
 properties that were explicitly set to None.

 I'll keep you posted


 Cheers,
 Flavio

 [0] https://review.openstack.org/#/c/138184/


 Regards
 Malini

 -Original Message-
 From: Flavio Percoco [mailto:fla...@redhat.com]
 Sent: Monday, June 08, 2015 11:47 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Cc: openstack-operators@lists.openstack.org
 Subject: Re: [openstack-dev] [glance] [nova] Glance bug with Kilo
 upgrade  Nova

 On 08/06/15 11:46 -0400, Clayton O'Neill wrote:

 We tested testing Kilo upgrades in our hardware dev environments last
 week and the second time through ran into this bug which right now is
 probably a show-stopper for us.

 https://bugs.launchpad.net/glance/+bug/1419823

 The issue here is that the v1 Glance API allows you to create images
 with properties that are 'NULL' in the Glance database.  For example:

 glance image-create --name cirros_test --disk-format qcow2
 --container-format bare --file cirros-0.3.4-x86_64-disk.img
 --is-public True --is-protected True --progress --property
 description=

 It's apparently also fairly easy to end up with a NULL description
 when editing images properties via Horizon.

 The issue is that the v2 Glance API returns these NULL properties to
 the client, which then validates them against the schema returned by
 the v2 API.
 This schema returns specifies that the description property *must* be
 a string.

 In the Kilo release, Nova has been changed to use the v2 API, so
 suddenly this matters.  The net effect is that end users can pretty
 easily create properties with NULL values, and then won't be able to
 boot instances using those images.
 What makes this worse is that it's completely opaque to end users,
 since this just reports that no node was available to schedule the
 instance.

 However, Nova *only* uses the v2 api to list images if the
 glance.allowed_direct_url_schemes config key is set in the config file.
 However, this config item defaults to an empty array, meaning that by
 default it's *always* set.  There doesn't appear to be a way to unset
 a value with oslo-config that has a default value, blocking off that
 route to work around the issue.  Disabling the v2 Glance API we don't
 think will work, since Nova appears to assume the v2 API is available.


 AFAIK, Nova supports V2 image-lists since before Juno when the
 allowed_direct_url_schemes config option is set. Are you referring to
 another change? Has the default been changed in Nova? I'm asking because I
 was working on this migration to V2 and we decided to postpone it to L.


 Another work around we've looked at is to change the DB schema for
 image properties (yuck) to not allow NULL values.  This results in
 Glance returning a
 500 error since glance-api is attempting to insert an invalid value.
 This is better than instances failing in an opaque fashion, but still
 pretty horrible.

 Has anyone else run into this issue yet?  Are there other work
 arounds that we've not thought of other than Don't create images with
 NULL properties?
 User education is definitely an option, but given the failure mode,
 it's not a great solution for us.


 I believe this needs to be fixed in the client rather than the API
 and/or the schemas. I'll take a look at this right away.

 Another workaround could be updateting `schema-image.json` to add the
 schemas that are missing and let the client download the final

Re: [Openstack-operators] Mar 9-10, 2015, Philadelphia, USA - Next Ops Meetup

[Clayton O'Neill]

 Thanks Tom! Are there any details regarding hotel accommodations?


It's not official, but several us from Time Warner Cable are staying at
http://www.thewindsorsuites.com/ which appears to be fairly close and
reasonably priced for the area.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators