Re: [openstack-dev] [neutron] multinode CI jobs in the gate

[John Schwarz]

Hey,

There is already an experimental job called
"gate-tempest-dsvm-neutron-dvr-ha-multinode-full-ubuntu-xenial-nv"
(which was added by [1]).

However, the job is currently a "one dvr_snat nodes, two dvr nodes"
deployment (instead of the "two dvr_snat nodes, one dvr node" that it
should be in order to make L3HA work there). The patch which makes the
final transition is being worked on in [2], and it's ready to be
merged afaik. Once it merges, we'll have a DVR+HA gate in the
experimental queue.

John.

[1]: https://review.openstack.org/#/c/383742/
[2]: https://review.openstack.org/#/c/383827/

On Fri, Dec 16, 2016 at 2:23 AM, Armando M.  wrote:
> Hi Neutrinos,
>
> Infra patch proposed in [1] got me thinking again about what we shall do
> when it comes to multinode testing in the gate and how to focus our testing
> and CI efforts upstream going forward. My line of thinking has always been
> that multinode resources should be devoted to configurations whose coverage
> fully benefit from the inherent nature of the distributed deployment, and
> that means giving priority to DVR+HA and demote other configurations that
> use centralized routing.
>
> I know that some of you in team have worked on this, but I am a bit behind
> on the latest status update. Any good soul willing to bring a strapped (for
> time) PTL up to speed?
>
> Cheers,
> Armando
>
> [1] https://review.openstack.org/#/c/411263/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
John Schwarz,
Senior Software Engineer,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] bug deputy report

[John Schwarz]

On Mon, Oct 24, 2016 at 7:39 AM, Das, Anindita  wrote:
> 1.   https://bugs.launchpad.net/neutron/+bug/1635554 - Delete Router /
> race condition
>

I've triaged this bug and assigned it to myself. I'll make sure this
either gets solved or gets the proper attention.

-- 
John Schwarz,
Senior Software Engineer,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Neutron team social event in Barcelona

[John Schwarz]

+1 - great initiative, can't wait to re-meet everybody :)

On Sat, Oct 15, 2016 at 8:11 PM, Paul Michali  wrote:
> +1 Thanks for setting this up!
>
>
> On Sat, Oct 15, 2016, 9:53 AM Alonso Hernandez, Rodolfo
>  wrote:
>>
>> +1
>>
>>
>>
>> From: Doug Wiegley [mailto:doug...@parksidesoftware.com]
>> Sent: Saturday, October 15, 2016 1:57 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> 
>> Subject: Re: [openstack-dev] [Neutron] Neutron team social event in
>> Barcelona
>>
>>
>>
>> +1
>>
>> Doug
>>
>>
>> On Oct 14, 2016, at 6:24 PM, Kevin Benton  wrote:
>>
>> +1
>>
>>
>>
>> On Oct 14, 2016 1:33 PM, "Miguel Lavalle"  wrote:
>>
>> Dear Neutrinos,
>>
>> I am organizing a social event for the team on Thursday 27th at 19:30.
>> After doing some Google research, I am proposing Raco de la Vila, which is
>> located in Poblenou: http://www.racodelavila.com/en/index.htm. The menu is
>> here: http://www.racodelavila.com/en/carta-racodelavila.htm
>>
>> It is easy to get there by subway from the Summit venue:
>> https://goo.gl/maps/HjaTEcBbDUR2. I made a reservation for 25 people under
>> 'Neutron' or "Miguel Lavalle". Please confirm your attendance so we can get
>> a final count.
>>
>> Here's some reviews:
>> https://www.tripadvisor.com/Restaurant_Review-g187497-d1682057-Reviews-Raco_De_La_Vila-Barcelona_Catalonia.html
>>
>>
>> Cheers
>>
>>
>>
>> Miguel
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
John Schwarz,
Senior Software Engineer,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Skip-only tests

[John Schwarz]

Hi all,

The Neutron community has recently started contributing Tempest
scenario tests in the Neutron tree and I'd like to discuss a general
issue we're hitting. Mainly, we're talking about required hardware for
some features and/or VM images that support more features, which makes
it hard (or impossible) to test certain features at the gate
end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and
tests relating to VLAN Aware VMs.

For this reason, we would like to bring up the idea of introducing in
a new concept to the "testing arena" - skip-only tests. These will be
tests that will be merged upstream and skipped unless some
pre-conditions are met (example: "@skip_unless(has_sriov=True)"). The
key idea here though, is that we don't expect upstream gates to add
support for these pre-conditions any time soon. Instead, the tests
will be merged upstream and used on the various downstream gates most
of us have (with specific hardware/deployment that match our
respective interests).

Another way to look at it is: "I want to contribute tests that require
specific hardware/software/deployments, and I want it to be robust and
shared amongst the community as per the Open Source way, even though
it won't be actually run on the upstream gates". Obviously, we would
continue to investigate ways to "unskip" these kind of tests where
possible.

Examples where this would be extremely helpful include (but aren't limited to):

1. SR-IOV - Sure, the Mellanox CI currently tests SR-IOV in VF mode,
however this isn't enough; SR-IOV in PF mode isn't checked and
additional interesting use-cases (such as QoS + SR-IOV) aren't tested
in Mellanox' CI (let alone any other upstream gate). Also, considering
the current QoS test - in order to test it specifically with SR-IOV,
one must be able to create a port with vnic_type='direct'.

2. VLAN Aware VMs - while tested (or planned to be tested) upstream in
the unit/functional/api/fullstack levels, it isn't tested (nor is it
planned to be, afaik) in a tempest scenario. The reason is simple: VMs
require 802.1Q [1] in order to support VLAN Aware VMs, and it isn't
currently in the Cirros image. While we're currently blocking on these
tests until the Cirros images are updated, we could easily test the
feature using our respective images downstream. This is also a good
example where eventually these tests can be un-skipped when the
upstream Cirros images are updated.

3. SCTP tests for security groups - also blocked because Cirros
doesn't carry a netcat which supports this.

...And so on and so forth.


Here's a brief QA section that should include some common questions/complaints:

Q: The VLAN Aware VMs issue can be solved by adding Cirros support for
802.1Q - why should we in the mean time merge tests that need to be
maintained?
A: No one has guarantees when the new Cirros will be released and that
it will work with all the other gate jobs and tests. Though we have
already started pushing for a new version with support, we have
(downstream-wise) other VMs that do carry the support required - we
are just missing the tests.

Q: So write the tests downstream - why should I care?
A: We want to collaborate on the tests upstream so we (and you) can
gain the standard benefits of improved and better maintained code.
This will also (hopefully) help removing duplication of efforts in the
testing scene.

Q: So how will we know if a downstream test fails?
A: This is a currently open question. One suggestion we had was that
any time these skip-only tests are explicitly changed, the author has
to show logs of a successful run. While we don't see how this can be
relevant to certain trivial project-wide changes (and such changes can
be exempt from such proofs), explicit modification of tests, logic or
infra that involve these tests will need to prove that they actually
work downstream.


We are now open for ideas about this idea.

[1]: https://en.wikipedia.org/wiki/IEEE_802.1Q

-- 
John Schwarz,
Senior Software Engineer,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Newton Midcycle - Tuesday dinner

[John Schwarz]

Hi guys,

For those of us who'll arrive on Tuesday to Cork, Martin Hickey has
arranged a dinner at "Gourmet Burger Bistro" [1], 8 Bridge Street, at
19:30. Last I heard the reservation was for 15 people so this should
accommodate all who filled out in [2] that they will arrive on
Tuesday.

[1]: http://www.gourmetburgerbistro.ie/
[2]: https://etherpad.openstack.org/p/newton-neutron-midcycle

See you then,

-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [tooz] DLM benchmark results

[John Schwarz]

Yes, the backends were deployed in cluster configuration (the
configurations are available in the appendix).
I'll make a change to the doc to make sure this is reflected properly.

On Fri, Jul 22, 2016 at 11:29 AM, Kevin Benton  wrote:
> Were the backends (zookeeper, etcd) deployed in a cluster configuration? I
> can't quite tell from the doc.
>
> On Fri, Jul 22, 2016 at 12:58 AM, John Schwarz  wrote:
>>
>> You're right Joshua.
>>
>> Tooz HEAD points to 0f4e1198fdcbd6a29d77c67d105d201ed0fbd9e0.
>>
>> With regards to etcd and zookeeper's versions, they are:
>> zookeeper-3.4.5+28-1.cdh4.7.1.p0.13.el6.x86_64,
>> etcd-2.2.5-2.el7.0.1.x86_64.
>>
>> John.
>>
>> On Thu, Jul 21, 2016 at 8:14 PM, Joshua Harlow 
>> wrote:
>> > Hi John,
>> >
>> > Thanks for gathering this info,
>> >
>> > Do you have the versions of the backend that were used here
>> > (particularly
>> > relevant for etcd which has a new release pretty frequently).
>> >
>> > It'd be useful to capture that info also :)
>> >
>> > John Schwarz wrote:
>> >>
>> >> Hi everyone,
>> >>
>> >> Following [1], a few of us sat down during the last day of the Austin
>> >> Summit and discussed the possibility of adding formal support for
>> >> Tooz, specifically for the locking mechanism it provides. The
>> >> conclusion we reached was that benchmarks should be done to show if
>> >> and how Tooz affects the normal operation of Neutron (i.e. if locking
>> >> a resource using Zookeeper takes 3 seconds, it's not worthwhile at
>> >> all).
>> >>
>> >> We've finally finished the benchmarks and they are available at [2].
>> >> They test a specific case: when creating an HA router a lock-free
>> >> algorithm is used to assign a vrid to a router (this is later used for
>> >> keepalived), and the benchmark specifically checks the effects of
>> >> locking that function with either Zookeeper or Etcd, using the no-Tooz
>> >> case as a baseline. The locking was checked in 2 different ways - one
>> >> which presents no contention (acquire() always succeeds immediately)
>> >> and one which presents contentions (acquire() may block until a
>> >> similar process for the invoking tenant is complete).
>> >>
>> >> The benchmarks show that while using Tooz does raise the cost of an
>> >> operation, the effects are not as bad as we initially feared. In the
>> >> simple, single simultaneous request, using Zookeeper raised the
>> >> average time it took to create a router by 1.5% (from 11.811 to 11.988
>> >> seconds). On the more-realistic case of 6 simultaneous requests,
>> >> Zookeeper raised the cost by 3.74% (from 16.533 to 17.152 seconds).
>> >>
>> >> It is important to note that the setup itself was overloaded - it was
>> >> built on a single baremetal hosting 5 VMs (4 of which were
>> >> controllers) and thus we were unable to go further - for example, 10
>> >> concurrent requests overloaded the server and caused some race
>> >> conditions to appear in the L3 scheduler (bugs will be opened soon),
>> >> so for this reason we haven't tested heavier samples and limited
>> >> ourselves to 6 simultaneous requests.
>> >>
>> >> Also important to note that some kind of race condition was noticed in
>> >> tooz's etcd driver. We've discussed this with the tooz devs and
>> >> provided a patch that is supposed to fix them [3].
>> >> Lastly, races in the L3 HA Scheduler were found and we are yet to dig
>> >> into them and find out their cause - bugs will be opened for these as
>> >> well.
>> >>
>> >> I've opened the summary [2] for comments so you're welcome to open a
>> >> discussion about the results both in the ML and on the doc itself.
>> >>
>> >> (CC to all those who attended the Austin Summit meeting and other
>> >> interested parties)
>> >> Happy locking,
>> >>
>> >> [1]:
>> >>
>> >> http://lists.openstack.org/pipermail/openstack-dev/2016-April/093199.html
>> >> [2]:
>> >>
>> >> https://docs.google.com/document/d/1jdI8gkQKBE0G9koR0nLiW02d5rwyWv_-gAp7yavt4w8
>> >> [3]: https://review.openstack.org/#/c/342096/
>> >>
>> >> --
>> >> Joh

Re: [openstack-dev] [neutron] [tooz] DLM benchmark results

[John Schwarz]

You're right Joshua.

Tooz HEAD points to 0f4e1198fdcbd6a29d77c67d105d201ed0fbd9e0.

With regards to etcd and zookeeper's versions, they are:
zookeeper-3.4.5+28-1.cdh4.7.1.p0.13.el6.x86_64,
etcd-2.2.5-2.el7.0.1.x86_64.

John.

On Thu, Jul 21, 2016 at 8:14 PM, Joshua Harlow  wrote:
> Hi John,
>
> Thanks for gathering this info,
>
> Do you have the versions of the backend that were used here (particularly
> relevant for etcd which has a new release pretty frequently).
>
> It'd be useful to capture that info also :)
>
> John Schwarz wrote:
>>
>> Hi everyone,
>>
>> Following [1], a few of us sat down during the last day of the Austin
>> Summit and discussed the possibility of adding formal support for
>> Tooz, specifically for the locking mechanism it provides. The
>> conclusion we reached was that benchmarks should be done to show if
>> and how Tooz affects the normal operation of Neutron (i.e. if locking
>> a resource using Zookeeper takes 3 seconds, it's not worthwhile at
>> all).
>>
>> We've finally finished the benchmarks and they are available at [2].
>> They test a specific case: when creating an HA router a lock-free
>> algorithm is used to assign a vrid to a router (this is later used for
>> keepalived), and the benchmark specifically checks the effects of
>> locking that function with either Zookeeper or Etcd, using the no-Tooz
>> case as a baseline. The locking was checked in 2 different ways - one
>> which presents no contention (acquire() always succeeds immediately)
>> and one which presents contentions (acquire() may block until a
>> similar process for the invoking tenant is complete).
>>
>> The benchmarks show that while using Tooz does raise the cost of an
>> operation, the effects are not as bad as we initially feared. In the
>> simple, single simultaneous request, using Zookeeper raised the
>> average time it took to create a router by 1.5% (from 11.811 to 11.988
>> seconds). On the more-realistic case of 6 simultaneous requests,
>> Zookeeper raised the cost by 3.74% (from 16.533 to 17.152 seconds).
>>
>> It is important to note that the setup itself was overloaded - it was
>> built on a single baremetal hosting 5 VMs (4 of which were
>> controllers) and thus we were unable to go further - for example, 10
>> concurrent requests overloaded the server and caused some race
>> conditions to appear in the L3 scheduler (bugs will be opened soon),
>> so for this reason we haven't tested heavier samples and limited
>> ourselves to 6 simultaneous requests.
>>
>> Also important to note that some kind of race condition was noticed in
>> tooz's etcd driver. We've discussed this with the tooz devs and
>> provided a patch that is supposed to fix them [3].
>> Lastly, races in the L3 HA Scheduler were found and we are yet to dig
>> into them and find out their cause - bugs will be opened for these as
>> well.
>>
>> I've opened the summary [2] for comments so you're welcome to open a
>> discussion about the results both in the ML and on the doc itself.
>>
>> (CC to all those who attended the Austin Summit meeting and other
>> interested parties)
>> Happy locking,
>>
>> [1]:
>> http://lists.openstack.org/pipermail/openstack-dev/2016-April/093199.html
>> [2]:
>> https://docs.google.com/document/d/1jdI8gkQKBE0G9koR0nLiW02d5rwyWv_-gAp7yavt4w8
>> [3]: https://review.openstack.org/#/c/342096/
>>
>> --
>> John Schwarz,
>> Senior Software Engineer,
>> Red Hat.
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
John Schwarz,
Senior Software Engineer,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] [tooz] DLM benchmark results

[John Schwarz]

Hi everyone,

Following [1], a few of us sat down during the last day of the Austin
Summit and discussed the possibility of adding formal support for
Tooz, specifically for the locking mechanism it provides. The
conclusion we reached was that benchmarks should be done to show if
and how Tooz affects the normal operation of Neutron (i.e. if locking
a resource using Zookeeper takes 3 seconds, it's not worthwhile at
all).

We've finally finished the benchmarks and they are available at [2].
They test a specific case: when creating an HA router a lock-free
algorithm is used to assign a vrid to a router (this is later used for
keepalived), and the benchmark specifically checks the effects of
locking that function with either Zookeeper or Etcd, using the no-Tooz
case as a baseline. The locking was checked in 2 different ways - one
which presents no contention (acquire() always succeeds immediately)
and one which presents contentions (acquire() may block until a
similar process for the invoking tenant is complete).

The benchmarks show that while using Tooz does raise the cost of an
operation, the effects are not as bad as we initially feared. In the
simple, single simultaneous request, using Zookeeper raised the
average time it took to create a router by 1.5% (from 11.811 to 11.988
seconds). On the more-realistic case of 6 simultaneous requests,
Zookeeper raised the cost by 3.74% (from 16.533 to 17.152 seconds).

It is important to note that the setup itself was overloaded - it was
built on a single baremetal hosting 5 VMs (4 of which were
controllers) and thus we were unable to go further - for example, 10
concurrent requests overloaded the server and caused some race
conditions to appear in the L3 scheduler (bugs will be opened soon),
so for this reason we haven't tested heavier samples and limited
ourselves to 6 simultaneous requests.

Also important to note that some kind of race condition was noticed in
tooz's etcd driver. We've discussed this with the tooz devs and
provided a patch that is supposed to fix them [3].
Lastly, races in the L3 HA Scheduler were found and we are yet to dig
into them and find out their cause - bugs will be opened for these as
well.

I've opened the summary [2] for comments so you're welcome to open a
discussion about the results both in the ML and on the doc itself.

(CC to all those who attended the Austin Summit meeting and other
interested parties)
Happy locking,

[1]: http://lists.openstack.org/pipermail/openstack-dev/2016-April/093199.html
[2]: 
https://docs.google.com/document/d/1jdI8gkQKBE0G9koR0nLiW02d5rwyWv_-gAp7yavt4w8
[3]: https://review.openstack.org/#/c/342096/

--
John Schwarz,
Senior Software Engineer,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

[John Schwarz]

The incorporation of tooz and Neutron is being discussed as part of
https://bugs.launchpad.net/neutron/+bug/1552680 as an RFE for Newton.
Hopefully I'll have some news to break in on this matter in the
upcoming days (and if I do I'll update on the launchpad to eliminate
duplicity).

On Tue, May 24, 2016 at 8:54 AM, Gary Kotton  wrote:
> Hi,
>
> We have used tooz to enable concurrency. Zookeeper and Redis worked well. I
> think that it is certainly something that we need to consider. The challenge
> becomes a deployment.
>
> Thanks
>
> Gary
>
>
>
> From: Damon Wang 
> Reply-To: OpenStack List 
> Date: Tuesday, May 24, 2016 at 5:58 AM
> To: OpenStack List 
> Subject: Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency
> issues
>
>
>
> Hi,
>
>
>
> I want to add an option which handle by another project Tooz.
>
>
>
> https://github.com/openstack/tooz
>
>
>
> with redis or some other drivers, it seems pretty a good choice.
>
>
>
> Any comments?
>
>
>
> Wei Wang
>
>
>
> 2016-05-17 6:53 GMT+08:00 Ilya Chukhnakov :
>
>
>
> On 16 May 2016, at 20:01, Michał Dulko  wrote:
>
>
> It's not directly related, but this reminds me of tests done by geguileo
> [1] some time ago that were comparing different methods of preventing DB
> race conditions in concurrent environment. Maybe you'll also find them
> useful as you'll probably need to do something like conditional update
> to increment a revision number.
>
> [1] https://github.com/Akrog/test-cinder-atomic-states
>
>
>
> Thanks for the link. The SQLA revisions are similar to the
> 'solutions/update_with_where',
>
> but they use the dedicated column for that [2]. And as long as it is
> properly configured,
>
> it happens 'automagically' (SQLA will take care of adding proper 'where' to
> 'update').
>
>
>
> [2] http://docs.sqlalchemy.org/en/latest/orm/versioning.html
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Discussing DLMs in the Unplugged Track

[John Schwarz]

Hi guys,

We'll meet at 11:00 at Salon C, Hilton.

John.

On Sun, Apr 24, 2016 at 8:59 AM, Joshua Harlow  wrote:
> I'll try to be there as well, since I know a little bit about tooz ;)
>
> -Josh
>
>
> Gary Kotton wrote:
>>
>> Hi,
>> I suggest that you speak with Kobi Samoray - he implemented this for the
>> vmware_nsx repository using tooz. Its pretty cool.
>> Thanks
>> Gary
>>
>>
>>
>>
>> On 4/23/16, 5:16 PM, "John Schwarz"  wrote:
>>
>>> Hi guys,
>>>
>>> I'm interested in discussing the DLM RFE [1] during the unplugged
>>> track on Friday morning, around 11:00am. A face-to-face talk about
>>> this with all parties interested will surely prove fruitful and will
>>> set us up on the track we want to go through in respect to this
>>> feature.
>>>
>>> You can find past discussions about this topic in previous driver
>>> meetings here: [2], [3].
>>>
>>> If you're interested in this topic, please come to talk :)
>>>
>>> [1]: https://bugs.launchpad.net/neutron/+bug/1552680
>>> [2]:
>>> http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-03-31-22.00.log.html#l-231
>>> [3]:
>>> http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-04-14-22.00.log.html#l-153
>>>
>>> --
>>> John Schwarz,
>>> Red Hat.
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Social at the summit

[John Schwarz]

+1
On 25 Apr 2016 16:48, "Carl Baldwin"  wrote:

> +1
> On Apr 25, 2016 10:57 AM, "Kyle Mestery"  wrote:
>
>> Ihar, Henry and I were talking and we thought Thursday night makes sense
>> for a Neutron social in Austin. If others agree, reply on this thread and
>> we'll find a place.
>>
>> Thanks!
>> Kyle
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Discussing DLMs in the Unplugged Track

[John Schwarz]

Hi guys,

I'm interested in discussing the DLM RFE [1] during the unplugged
track on Friday morning, around 11:00am. A face-to-face talk about
this with all parties interested will surely prove fruitful and will
set us up on the track we want to go through in respect to this
feature.

You can find past discussions about this topic in previous driver
meetings here: [2], [3].

If you're interested in this topic, please come to talk :)

[1]: https://bugs.launchpad.net/neutron/+bug/1552680
[2]: 
http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-03-31-22.00.log.html#l-231
[3]: 
http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-04-14-22.00.log.html#l-153

-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] L3 HA testing on scale

[John Schwarz]

This is some awesome work, Ann. It's very neat to see that all the
races we've struggled with w.r.t. the l3 scheduler has paid off. I
would definitely like to see how these results are effected by
https://review.openstack.org/#/c/305774/ but understandably 49
physical nodes are hard to come by.

Also, we should see how to best handle of the issue Ann found (and is
tracked at https://review.openstack.org/#/c/305774/). Specifically,
reproducing this should be our goal.

John.

On Mon, Apr 18, 2016 at 5:15 PM, Anna Kamyshnikova
 wrote:
> Hi guys!
>
> As a developer I use Devstack or multinode OpenStack installation (4-5
> nodes) for work, but these are "abstract" environments, where you are not
> able to perform some scenarios as your machine is not powerful enough. But
> it is really important to understand the issues that real deployments have.
>
> Recently I've performed testing of L3 HA on the scale environment 49 nodes
> (3 controllers, 46 computes) Fuel 8.0. On this environment I ran shaker and
> rally tests and also performed some manual destructive scenarios. I think
> that this is very important to share these results. Ideally, I think that we
> should collect statistics for different configurations each release to
> compare and check it to make sure that we are heading the right way.
>
> The results of shaker and rally tests [1]. I put detailed report in google
> doc [2]. I would appreciate all comments on these results.
>
> [1] - http://akamyshnikova.github.io/neutron-benchmark-results/
> [2] -
> https://docs.google.com/a/mirantis.com/document/d/1TFEUzRRlRIt2HpsOzFh-RqWwgTzJPBefePPA0f0x9uw/edit?usp=sharing
>
> Regards,
> Ann Kamyshnikova
> Mirantis, Inc
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Newton Design summit schedule - Draft

[John Schwarz]

Hi guys,

Note that the wiki page's timestamps<->session title was a bit
outdated so I've corrected where I've seen the discrepancies.
Specifically the "future of Neutron architecture" and "future of
Neutron client" were swapped.

John.

On Tue, Apr 12, 2016 at 9:47 PM, Armando M.  wrote:
>
>
> On 12 April 2016 at 07:08, Michael Johnson  wrote:
>>
>> Armando,
>>
>> Is there any way we can move the "Neutron: Development track: future
>> of *-aas projects" session?  I overlaps with the LBaaS talk:
>>
>> https://www.openstack.org/summit/austin-2016/summit-schedule/events/6893?goback=1
>>
>> Michael
>
>
> Swapped with the first slot of the day. I also loaded etherpads here:
>
> https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#Neutron
>
> Cheers,
> Armando
>>
>>
>>
>> On Mon, Apr 11, 2016 at 9:56 PM, Armando M.  wrote:
>> > Hi folks,
>> >
>> > A provisional schedule for the Neutron project is available [1]. I am
>> > still
>> > working with the session chairs and going through/ironing out some
>> > details
>> > as well as gathering input from [2].
>> >
>> > I hope I can get something more final by the end of this week. In the
>> > meantime, please free to ask questions/provide comments.
>> >
>> > Many thanks,
>> > Armando
>> >
>> > [1]
>> >
>> > https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Neutron%3A
>> > [2] https://etherpad.openstack.org/p/newton-neutron-summit-ideas
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] client noauth deprecation

[John Schwarz]

Adding to what Miguel said, I've merged a few months back a patch [1]
which actually fixed the noauth feature in favour of the full-stack
effort (the full-stack patches use neutronclient in noauth mode for easy
access to neutron-server). We'll probably continue to use neutronclient
until some other alternative is mature enough (for example, API testing
providing thorough interface to neutron-server).

Needless to say that noauth is currently working just fine and if it
were to stop working I'll be very sad ;-)

[1]: https://review.openstack.org/#/c/125022/

On 01/07/2015 11:44 AM, Miguel Ángel Ajo wrote:
> Seems like a good reason to keep it, this allows us to test 
> internal integration in isolation from keystone.
> 
> Miguel Ángel Ajo
> 
> On Wednesday, 7 de January de 2015 at 10:05, Assaf Muller wrote:
> 
>>
>>
>> - Original Message -
>>> The option to disable keystone authentication in the neutron client was
>>> marked for deprecation in August as part of a Keystone support
>>> upgrade.[1]
>>>
>>> What was the reason for this? As far as I can tell, Neutron works
>>> fine in the
>>> 'noauth' mode and there isn't a lot of code that tightly couples
>>> neutron to
>>> Keystone that I can think of.
>>
>> It was actually broken until John fixed it in:
>> https://review.openstack.org/#/c/125022/
>>
>> We plan on using it in the Neutron in-tree full-stack testing. I'd
>> appreciate
>> if the functionality was not removed or otherwise broken :)
>>
>>>
>>> 1.
>>> https://github.com/openstack/python-neutronclient/commit/2203b013fb66808ef280eff0285318ce21d9bc67#diff-ba2e4fad85e66d9aabb6193f222fcc4cR438
>>>
>>> --
>>> Kevin Benton
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][Spec freeze exception] Rootwrap daemon ode support

[John Schwarz]

+1. I think this is an important feature and I'd be happy to do reviews
when needed - hopefully this can get in by Kilo! :)

On 11/08/2014 12:24 PM, Armando M. wrote:
> Hi Miguel,
> 
> Thanks for picking this up. Pull me in and I'd be happy to help!
> 
> Cheers,
> Armando
> 
> On 7 November 2014 10:05, Miguel Ángel Ajo  > wrote:
> 
> 
> Hi Yorik,
> 
>I was talking with Mark Mcclain a minute ago here at the summit
> about this. And he told me that now at the start of the cycle looks
> like a good moment to merge the spec & the root wrap daemon bits, so
> we have a lot of headroom for testing during the next months.
> 
>We need to upgrade the spec [1] to the new Kilo format.
> 
>Do you have some time to do it?, I can allocate some time and do
> it right away.
> 
> [1] https://review.openstack.org/#/c/93889/
> -- 
> Miguel Ángel Ajo
> Sent with Sparrow 
> 
> On Thursday, 24 de July de 2014 at 01:42, Miguel Angel Ajo Pelayo wrote:
> 
>> +1
>>
>> Sent from my Android phone using TouchDown (www.nitrodesk.com
>> )
>>
>>
>> -Original Message-
>> From: Yuriy Taraday [yorik@gmail.com
>> ]
>> Received: Thursday, 24 Jul 2014, 0:42
>> To: OpenStack Development Mailing List
>> [openstack-dev@lists.openstack.org
>> ]
>> Subject: [openstack-dev] [Neutron][Spec freeze exception] Rootwrap
>> daemon mode support
>>
>>
>> Hello.
>>
>> I'd like to propose making a spec freeze exception for
>> rootwrap-daemon-mode spec [1].
>>
>> Its goal is to save agents' execution time by using daemon mode
>> for rootwrap and thus avoiding python interpreter startup time as
>> well as sudo overhead for each call. Preliminary benchmark shows
>> 10x+ speedup of the rootwrap interaction itself.
>>
>> This spec have a number of supporters from Neutron team (Carl and
>> Miguel gave it their +2 and +1) and have all code waiting for
>> review [2], [3], [4].
>> The only thing that has been blocking its progress is Mark's -2
>> left when oslo.rootwrap spec hasn't been merged yet. Now that's
>> not the case and code in oslo.rootwrap is steadily getting
>> approved [5].
>>
>> [1] https://review.openstack.org/93889
>> [2] https://review.openstack.org/82787
>> [3] https://review.openstack.org/84667
>> [4] https://review.openstack.org/107386
>> [5] 
>> https://review.openstack.org/#/q/project:openstack/oslo.rootwrap+topic:bp/rootwrap-daemon-mode,n,z
>>
>> -- 
>>
>> Kind regards, Yuriy.
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] non-deterministic gate failures due to unclosed eventlet Timeouts

[John Schwarz]

Hi,

Long story short: for future reference, if you initialize an eventlet
Timeout, make sure you close it (either with a context manager or simply
timeout.close()), and be extra-careful when writing tests using
eventlet Timeouts, because these timeouts don't implicitly expire and
will cause unexpected behaviours (see [1]) like gate failures. In our
case this caused non-deterministic failures on the dsvm-functional test
suite.


Late last week, a bug was found ([2]) in which an eventlet Timeout
object was initialized but not closed. This instance was left inside
eventlet's inner-workings and triggered non-deterministic "Timeout: 10
seconds" errors and failures in dsvm-functional tests.

As mentioned earlier, initializing a new eventlet.timeout.Timeout
instance also registers it to inner mechanisms that exist within the
library, and the reference remains there until it is explicitly removed
(and not until the scope leaves the function block, as some would have
thought). Thus, the old code (simply creating an instance without
assigning it to a variable) left no way to close the timeout object.
This reference remains throughout the "life" of a worker, so this can
(and did) effect other tests and procedures using eventlet under the
same process. Obviously this could easily effect production-grade
systems with very high load.

For future reference:
 1) If you run into a "Timeout: %d seconds" exception whose traceback
includes "hub.switch()" and "self.greenlet.switch()" calls, there might
be a latent Timeout somewhere in the code, and a search for all
eventlet.timeout.Timeout instances will probably produce the culprit.

 2) The setup used to reproduce this error for debugging purposes is a
baremetal machine running a VM with devstack. In the baremetal machine I
used some 6 "dd if=/dev/zero of=/dev/null" to simulate high CPU load
(full command can be found at [3]), and in the VM I ran the
dsvm-functional suite. Using only a VM with similar high CPU simulation
fails to produce the result.

[1]
http://eventlet.net/doc/modules/timeout.html#eventlet.timeout.eventlet.timeout.Timeout.Timeout.cancel
[2] https://review.openstack.org/#/c/119001/
[3]
http://stackoverflow.com/questions/2925606/how-to-create-a-cpu-spike-with-a-bash-command


--
John Schwarz,
Software Engineer, Red Hat.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] [LBaaS] LBaaS v2 API syntax additions/changes

[John Schwarz]

On 08/25/2014 10:06 PM, Brandon Logan wrote:
>>
>> 2. Therefor, there should be some configuration to specifically enable
>> either version (not both) in case LBaaS is needed. In this case, the
>> other version is disabled (ie. a REST query for non-active version
>> should return a "not activated" error). Additionally, adding a
>> 'lb-version' command to return the version currently active seems like a
>> good user-facing idea. We should see how this doesn't negatively effect
>> the db migration process (for example, allowing read-only commands for
>> both versions?)
> 
> A /version endpoint can be added for both v1 and v2 extensions and
> service plugins.  If it doesn't already exist, it would be nice if
> neutron had an endpoint that would return the list of loaded extensions
> and their versions.
> 
There is 'neutron ext-list', but I'm not familiar enough with it or with
the REST API to say if we can use that.
>>
>> 3. Another decision that's needed to be made is the syntax for v2. As
>> mentioned, the current new syntax is 'neutron lbaas--'
>> (against the old 'lb--'), keeping in mind that once v1
>> is deprecated, a syntax like 'lbv2--' would be probably
>> unwanted. Is 'lbaas--' okay with everyone?
> 
> That is the reason we with with lbaas because lbv2 looks ugly and we'd
> be stuck with it for the lifetime of v2, unless we did another migration
> back to lb for it.  Which seemed wrong to do, since then we'd have to
> accept both lbv2 and lb commands, and then deprecate lbv2.
> 
> I assume this also means you are fine with the prefix in the API
> resource of /lbaas as well then?
> 
I don't mind, as long there is a similar mechanism which disables the
non-active REST API commands. Does anyone disagree?
>>
>> 4. If we are going for different API between versions, appropriate
>> patches also need to be written for lbaas-related scripts and also
>> Tempest, and their maintainers should probably be notified.
> 
> Could you elaborate on this? I don't understand what you mean by
> "different API between version."
> 
The intention was that the change of the user-facing API also forces
changes on other levels - not only neutronclient needs to be modified
accordingly, but also tempest system tests, horizon interface regarding
LBaaS...

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] [LBaaS] LBaaS v2 API syntax additions/changes

[John Schwarz]

Hi,

With the ongoing development of LBaaS v2, support for v2 of LBaaS in
neutronclient is also being developed, as can be seen in [1].
The current implementation adds a new syntax for v2; Whereas the v1
syntax is 'neutron lb--', the new v2 syntax is 'neutron
lbaas--'.

We fear that this can lead to some level of confusion on the users'
side. Currently, nothing prevents a user from trying to create a v1 pool
and then trying to add v2 members. Of course the second command will
fail, but the error message will be a non-intuitive one.

This was discussed at the last weekly IRC meeting ([2]). Some bulletins:

1. As was discussed in the hackathon, there shouldn't be more than one
version active at any given time - either v1 or v2. Also, using the same
syntax for both v1 and v2 is not a good idea (db migration will be
difficult when both version share syntax). We believe this should also
be enforced in the server side.

2. Therefor, there should be some configuration to specifically enable
either version (not both) in case LBaaS is needed. In this case, the
other version is disabled (ie. a REST query for non-active version
should return a "not activated" error). Additionally, adding a
'lb-version' command to return the version currently active seems like a
good user-facing idea. We should see how this doesn't negatively effect
the db migration process (for example, allowing read-only commands for
both versions?)

3. Another decision that's needed to be made is the syntax for v2. As
mentioned, the current new syntax is 'neutron lbaas--'
(against the old 'lb--'), keeping in mind that once v1
is deprecated, a syntax like 'lbv2--' would be probably
unwanted. Is 'lbaas--' okay with everyone?

4. If we are going for different API between versions, appropriate
patches also need to be written for lbaas-related scripts and also
Tempest, and their maintainers should probably be notified.

Are there any issues with one of the points raised above? Does anyone
see any other problems which we forgot to write down?

Thanks a lot,

John Schwarz, Yair Fried,
Redhat.

[1]: https://review.openstack.org/#/c/111475
[2]:
http://eavesdrop.openstack.org/meetings/neutron_lbaas/2014/neutron_lbaas.2014-08-21-14.00.log.html

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev