Re: [ovs-discuss] Open vSwitch balance-slb bond mode with two upstream switches
On Fri, Jul 19, 2019 at 7:46 AM Grzegorz wrote: > Hi Jason > > I have the same doubts - I would like to connect balance-slb in OpenStack > Rocky to two separate switches. > > I see you wrote this post a few years ago. > Have you figured it out finally? > > Thank you in advance for your reply > > BR > Grzegorz Juszczak > This only works if the two switches are clustered, i.e. two or more chassis running as a single logical switch. The issue is that balance-slb will only use one link for a MAC+VLAN, but when rebalancing happens the traffic may start using the other port. If the switch attached to the original port doesn't know that the traffic is now using the other port/different switch, then traffic to the OVS bond may be lost. You can use active-backup with OVS bonds and two switches, or attach a bond/team using a different bonding technique (Linux kernel bonds, teamd adapter teams, etc.) that supports disparate switches and then attach that bond to OVS. There aren't many good options for attaching to multiple switches, so I try to use switches that can be clustered when I'm not using a chassis switch. Look into teamd, it has more options for controlling how your traffic is load balanced, and you can choose your own hash combinations. I've configured OpenStack deployments with teams attached to OVS bridges for Neutron. The best documentation I've found so far is here, but you may need to adjust depending on Linux distribution: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/ch-configure_network_teaming -- Dan Sneddon ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] ovn-controller Incremental Processing scale testing
On Fri, Jul 19, 2019 at 12:37 PM Numan Siddique wrote:
>
>
> On Fri, Jul 19, 2019 at 6:19 PM Numan Siddique
> wrote:
>
>>
>>
>> On Fri, Jul 19, 2019 at 6:28 AM Han Zhou wrote:
>>
>>>
>>>
>>> On Tue, Jul 9, 2019 at 12:13 AM Numan Siddique
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Tue, Jul 9, 2019 at 12:25 PM Daniel Alvarez Sanchez <
>>> dalva...@redhat.com> wrote:
>>> >>
>>> >> Thanks Numan for running these tests outside OpenStack!
>>> >>
>>> >> On Tue, Jul 9, 2019 at 7:50 AM Numan Siddique
>>> wrote:
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Tue, Jul 9, 2019 at 11:05 AM Han Zhou wrote:
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> On Fri, Jun 21, 2019 at 12:31 AM Han Zhou
>>> wrote:
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Thu, Jun 20, 2019 at 11:42 PM Numan Siddique <
>>> nusid...@redhat.com> wrote:
>>> >> >> > >
>>> >> >> > >
>>> >> >> > >
>>> >> >> > > On Fri, Jun 21, 2019, 11:47 AM Han Zhou
>>> wrote:
>>> >> >> > >>
>>> >> >> > >>
>>> >> >> > >>
>>> >> >> > >> On Tue, Jun 11, 2019 at 9:16 AM Daniel Alvarez Sanchez <
>>> dalva...@redhat.com> wrote:
>>> >> >> > >> >
>>> >> >> > >> > Thanks a lot Han for the answer!
>>> >> >> > >> >
>>> >> >> > >> > On Tue, Jun 11, 2019 at 5:57 PM Han Zhou
>>> wrote:
>>> >> >> > >> > >
>>> >> >> > >> > >
>>> >> >> > >> > >
>>> >> >> > >> > >
>>> >> >> > >> > > On Tue, Jun 11, 2019 at 5:12 AM Dumitru Ceara <
>>> dce...@redhat.com> wrote:
>>> >> >> > >> > > >
>>> >> >> > >> > > > On Tue, Jun 11, 2019 at 10:40 AM Daniel Alvarez Sanchez
>>> >> >> > >> > > > wrote:
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > Hi Han, all,
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > Lucas, Numan and I have been doing some 'scale'
>>> testing of OpenStack
>>> >> >> > >> > > > > using OVN and wanted to present some results and
>>> issues that we've
>>> >> >> > >> > > > > found with the Incremental Processing feature in
>>> ovn-controller. Below
>>> >> >> > >> > > > > is the scenario that we executed:
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > * 7 baremetal nodes setup: 3 controllers (running
>>> >> >> > >> > > > > ovn-northd/ovsdb-servers in A/P with pacemaker) + 4
>>> compute nodes. OVS
>>> >> >> > >> > > > > 2.10.
>>> >> >> > >> > > > > * The test consists on:
>>> >> >> > >> > > > > - Create openstack network (OVN LS), subnet and
>>> router
>>> >> >> > >> > > > > - Attach subnet to the router and set gw to the
>>> external network
>>> >> >> > >> > > > > - Create an OpenStack port and apply a Security
>>> Group (ACLs to allow
>>> >> >> > >> > > > > UDP, SSH and ICMP).
>>> >> >> > >> > > > > - Bind the port to one of the 4 compute nodes
>>> (randomly) by
>>> >> >> > >> > > > > attaching it to a network namespace.
>>> >> >> > >> > > > > - Wait for the port to be ACTIVE in Neutron ('up ==
>>> True' in NB)
>>> >> >> > >> > > > > - Wait until the test can ping the port
>>> >> >> > >> > > > > * Running browbeat/rally with 16 simultaneous process
>>> to execute the
>>> >> >> > >> > > > > test above 150 times.
>>> >> >> > >> > > > > * When all the 150 'fake VMs' are created, browbeat
>>> will delete all
>>> >> >> > >> > > > > the OpenStack/OVN resources.
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > We first tried with OVS/OVN 2.10 and pulled some
>>> results which showed
>>> >> >> > >> > > > > 100% success but ovn-controller is quite loaded (as
>>> expected) in all
>>> >> >> > >> > > > > the nodes especially during the deletion phase:
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > - Compute node: https://imgur.com/a/tzxfrIR
>>> >> >> > >> > > > > - Controller node (ovn-northd and ovsdb-servers):
>>> https://imgur.com/a/8ffKKYF
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > After conducting the tests above, we replaced
>>> ovn-controller in all 7
>>> >> >> > >> > > > > nodes by the one with the current master branch
>>> (actually from last
>>> >> >> > >> > > > > week). We also replaced ovn-northd and ovsdb-servers
>>> but the
>>> >> >> > >> > > > > ovs-vswitchd has been left untouched (still on 2.10).
>>> The expected
>>> >> >> > >> > > > > results were to get less ovn-controller CPU usage and
>>> also better
>>> >> >> > >> > > > > times due to the Incremental Processing feature
>>> introduced recently.
>>> >> >> > >> > > > > However, the results don't look very good:
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > - Compute node: https://imgur.com/a/wuq87F1
>>> >> >> > >> > > > > - Controller node (ovn-northd and ovsdb-servers):
>>> https://imgur.com/a/99kiyDp
>>> >> >> > >> > > > >
>>> >> >> > >> > > > > One thing that we can tell from the ovs-vswitchd CPU
>>> consumption is
>>> >> >> > >> > > > > that it's much less in the Incremental Processing
>>> (IP) case which
>>> >> >> > >> > > > > apparently doesn't make much sense. This led us to
>>> think that perhaps
>>> >> >> > >> > > > > ovn-controller was not installing the necessary flows
>>> in the switch
>>> >> >> > >> > > > > and we confirmed this hypothesis by looking into the
>>> dataplane
>>> >> >> > >> > > > > results. Out
Re: [ovs-discuss] [OVN] ovn-controller Incremental Processing scale testing
On Fri, Jul 19, 2019 at 6:19 PM Numan Siddique wrote:
>
>
> On Fri, Jul 19, 2019 at 6:28 AM Han Zhou wrote:
>
>>
>>
>> On Tue, Jul 9, 2019 at 12:13 AM Numan Siddique
>> wrote:
>> >
>> >
>> >
>> > On Tue, Jul 9, 2019 at 12:25 PM Daniel Alvarez Sanchez <
>> dalva...@redhat.com> wrote:
>> >>
>> >> Thanks Numan for running these tests outside OpenStack!
>> >>
>> >> On Tue, Jul 9, 2019 at 7:50 AM Numan Siddique
>> wrote:
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Jul 9, 2019 at 11:05 AM Han Zhou wrote:
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Fri, Jun 21, 2019 at 12:31 AM Han Zhou
>> wrote:
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Jun 20, 2019 at 11:42 PM Numan Siddique <
>> nusid...@redhat.com> wrote:
>> >> >> > >
>> >> >> > >
>> >> >> > >
>> >> >> > > On Fri, Jun 21, 2019, 11:47 AM Han Zhou
>> wrote:
>> >> >> > >>
>> >> >> > >>
>> >> >> > >>
>> >> >> > >> On Tue, Jun 11, 2019 at 9:16 AM Daniel Alvarez Sanchez <
>> dalva...@redhat.com> wrote:
>> >> >> > >> >
>> >> >> > >> > Thanks a lot Han for the answer!
>> >> >> > >> >
>> >> >> > >> > On Tue, Jun 11, 2019 at 5:57 PM Han Zhou
>> wrote:
>> >> >> > >> > >
>> >> >> > >> > >
>> >> >> > >> > >
>> >> >> > >> > >
>> >> >> > >> > > On Tue, Jun 11, 2019 at 5:12 AM Dumitru Ceara <
>> dce...@redhat.com> wrote:
>> >> >> > >> > > >
>> >> >> > >> > > > On Tue, Jun 11, 2019 at 10:40 AM Daniel Alvarez Sanchez
>> >> >> > >> > > > wrote:
>> >> >> > >> > > > >
>> >> >> > >> > > > > Hi Han, all,
>> >> >> > >> > > > >
>> >> >> > >> > > > > Lucas, Numan and I have been doing some 'scale'
>> testing of OpenStack
>> >> >> > >> > > > > using OVN and wanted to present some results and
>> issues that we've
>> >> >> > >> > > > > found with the Incremental Processing feature in
>> ovn-controller. Below
>> >> >> > >> > > > > is the scenario that we executed:
>> >> >> > >> > > > >
>> >> >> > >> > > > > * 7 baremetal nodes setup: 3 controllers (running
>> >> >> > >> > > > > ovn-northd/ovsdb-servers in A/P with pacemaker) + 4
>> compute nodes. OVS
>> >> >> > >> > > > > 2.10.
>> >> >> > >> > > > > * The test consists on:
>> >> >> > >> > > > > - Create openstack network (OVN LS), subnet and
>> router
>> >> >> > >> > > > > - Attach subnet to the router and set gw to the
>> external network
>> >> >> > >> > > > > - Create an OpenStack port and apply a Security
>> Group (ACLs to allow
>> >> >> > >> > > > > UDP, SSH and ICMP).
>> >> >> > >> > > > > - Bind the port to one of the 4 compute nodes
>> (randomly) by
>> >> >> > >> > > > > attaching it to a network namespace.
>> >> >> > >> > > > > - Wait for the port to be ACTIVE in Neutron ('up ==
>> True' in NB)
>> >> >> > >> > > > > - Wait until the test can ping the port
>> >> >> > >> > > > > * Running browbeat/rally with 16 simultaneous process
>> to execute the
>> >> >> > >> > > > > test above 150 times.
>> >> >> > >> > > > > * When all the 150 'fake VMs' are created, browbeat
>> will delete all
>> >> >> > >> > > > > the OpenStack/OVN resources.
>> >> >> > >> > > > >
>> >> >> > >> > > > > We first tried with OVS/OVN 2.10 and pulled some
>> results which showed
>> >> >> > >> > > > > 100% success but ovn-controller is quite loaded (as
>> expected) in all
>> >> >> > >> > > > > the nodes especially during the deletion phase:
>> >> >> > >> > > > >
>> >> >> > >> > > > > - Compute node: https://imgur.com/a/tzxfrIR
>> >> >> > >> > > > > - Controller node (ovn-northd and ovsdb-servers):
>> https://imgur.com/a/8ffKKYF
>> >> >> > >> > > > >
>> >> >> > >> > > > > After conducting the tests above, we replaced
>> ovn-controller in all 7
>> >> >> > >> > > > > nodes by the one with the current master branch
>> (actually from last
>> >> >> > >> > > > > week). We also replaced ovn-northd and ovsdb-servers
>> but the
>> >> >> > >> > > > > ovs-vswitchd has been left untouched (still on 2.10).
>> The expected
>> >> >> > >> > > > > results were to get less ovn-controller CPU usage and
>> also better
>> >> >> > >> > > > > times due to the Incremental Processing feature
>> introduced recently.
>> >> >> > >> > > > > However, the results don't look very good:
>> >> >> > >> > > > >
>> >> >> > >> > > > > - Compute node: https://imgur.com/a/wuq87F1
>> >> >> > >> > > > > - Controller node (ovn-northd and ovsdb-servers):
>> https://imgur.com/a/99kiyDp
>> >> >> > >> > > > >
>> >> >> > >> > > > > One thing that we can tell from the ovs-vswitchd CPU
>> consumption is
>> >> >> > >> > > > > that it's much less in the Incremental Processing (IP)
>> case which
>> >> >> > >> > > > > apparently doesn't make much sense. This led us to
>> think that perhaps
>> >> >> > >> > > > > ovn-controller was not installing the necessary flows
>> in the switch
>> >> >> > >> > > > > and we confirmed this hypothesis by looking into the
>> dataplane
>> >> >> > >> > > > > results. Out of the 150 VMs, 10% of them were
>> unreachable via ping
>> >> >> > >> > > > > when using ovn-controller from master.
>> >> >> > >> > > > >
>> >> >> > >> > > > > @Han, others, do you have any ideas
Re: [ovs-discuss] Open vSwitch balance-slb bond mode with two upstream switches
On Fri, Jul 19, 2019 at 12:12:31PM +0200, Grzegorz wrote: > I have the same doubts - I would like to connect balance-slb in OpenStack > Rocky to two separate switches. balance-slb doesn't work with that configuration, if I recall correctly. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Open vSwitch balance-slb bond mode with two upstream switches
Hi Jason I have the same doubts - I would like to connect balance-slb in OpenStack Rocky to two separate switches. I see you wrote this post a few years ago. Have you figured it out finally? Thank you in advance for your reply BR Grzegorz Juszczak ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] ovn-controller Incremental Processing scale testing
On Fri, Jul 19, 2019 at 6:28 AM Han Zhou wrote:
>
>
> On Tue, Jul 9, 2019 at 12:13 AM Numan Siddique
> wrote:
> >
> >
> >
> > On Tue, Jul 9, 2019 at 12:25 PM Daniel Alvarez Sanchez <
> dalva...@redhat.com> wrote:
> >>
> >> Thanks Numan for running these tests outside OpenStack!
> >>
> >> On Tue, Jul 9, 2019 at 7:50 AM Numan Siddique
> wrote:
> >> >
> >> >
> >> >
> >> > On Tue, Jul 9, 2019 at 11:05 AM Han Zhou wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Jun 21, 2019 at 12:31 AM Han Zhou wrote:
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Thu, Jun 20, 2019 at 11:42 PM Numan Siddique <
> nusid...@redhat.com> wrote:
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > On Fri, Jun 21, 2019, 11:47 AM Han Zhou
> wrote:
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >> On Tue, Jun 11, 2019 at 9:16 AM Daniel Alvarez Sanchez <
> dalva...@redhat.com> wrote:
> >> >> > >> >
> >> >> > >> > Thanks a lot Han for the answer!
> >> >> > >> >
> >> >> > >> > On Tue, Jun 11, 2019 at 5:57 PM Han Zhou
> wrote:
> >> >> > >> > >
> >> >> > >> > >
> >> >> > >> > >
> >> >> > >> > >
> >> >> > >> > > On Tue, Jun 11, 2019 at 5:12 AM Dumitru Ceara <
> dce...@redhat.com> wrote:
> >> >> > >> > > >
> >> >> > >> > > > On Tue, Jun 11, 2019 at 10:40 AM Daniel Alvarez Sanchez
> >> >> > >> > > > wrote:
> >> >> > >> > > > >
> >> >> > >> > > > > Hi Han, all,
> >> >> > >> > > > >
> >> >> > >> > > > > Lucas, Numan and I have been doing some 'scale' testing
> of OpenStack
> >> >> > >> > > > > using OVN and wanted to present some results and issues
> that we've
> >> >> > >> > > > > found with the Incremental Processing feature in
> ovn-controller. Below
> >> >> > >> > > > > is the scenario that we executed:
> >> >> > >> > > > >
> >> >> > >> > > > > * 7 baremetal nodes setup: 3 controllers (running
> >> >> > >> > > > > ovn-northd/ovsdb-servers in A/P with pacemaker) + 4
> compute nodes. OVS
> >> >> > >> > > > > 2.10.
> >> >> > >> > > > > * The test consists on:
> >> >> > >> > > > > - Create openstack network (OVN LS), subnet and router
> >> >> > >> > > > > - Attach subnet to the router and set gw to the
> external network
> >> >> > >> > > > > - Create an OpenStack port and apply a Security Group
> (ACLs to allow
> >> >> > >> > > > > UDP, SSH and ICMP).
> >> >> > >> > > > > - Bind the port to one of the 4 compute nodes
> (randomly) by
> >> >> > >> > > > > attaching it to a network namespace.
> >> >> > >> > > > > - Wait for the port to be ACTIVE in Neutron ('up ==
> True' in NB)
> >> >> > >> > > > > - Wait until the test can ping the port
> >> >> > >> > > > > * Running browbeat/rally with 16 simultaneous process
> to execute the
> >> >> > >> > > > > test above 150 times.
> >> >> > >> > > > > * When all the 150 'fake VMs' are created, browbeat
> will delete all
> >> >> > >> > > > > the OpenStack/OVN resources.
> >> >> > >> > > > >
> >> >> > >> > > > > We first tried with OVS/OVN 2.10 and pulled some
> results which showed
> >> >> > >> > > > > 100% success but ovn-controller is quite loaded (as
> expected) in all
> >> >> > >> > > > > the nodes especially during the deletion phase:
> >> >> > >> > > > >
> >> >> > >> > > > > - Compute node: https://imgur.com/a/tzxfrIR
> >> >> > >> > > > > - Controller node (ovn-northd and ovsdb-servers):
> https://imgur.com/a/8ffKKYF
> >> >> > >> > > > >
> >> >> > >> > > > > After conducting the tests above, we replaced
> ovn-controller in all 7
> >> >> > >> > > > > nodes by the one with the current master branch
> (actually from last
> >> >> > >> > > > > week). We also replaced ovn-northd and ovsdb-servers
> but the
> >> >> > >> > > > > ovs-vswitchd has been left untouched (still on 2.10).
> The expected
> >> >> > >> > > > > results were to get less ovn-controller CPU usage and
> also better
> >> >> > >> > > > > times due to the Incremental Processing feature
> introduced recently.
> >> >> > >> > > > > However, the results don't look very good:
> >> >> > >> > > > >
> >> >> > >> > > > > - Compute node: https://imgur.com/a/wuq87F1
> >> >> > >> > > > > - Controller node (ovn-northd and ovsdb-servers):
> https://imgur.com/a/99kiyDp
> >> >> > >> > > > >
> >> >> > >> > > > > One thing that we can tell from the ovs-vswitchd CPU
> consumption is
> >> >> > >> > > > > that it's much less in the Incremental Processing (IP)
> case which
> >> >> > >> > > > > apparently doesn't make much sense. This led us to
> think that perhaps
> >> >> > >> > > > > ovn-controller was not installing the necessary flows
> in the switch
> >> >> > >> > > > > and we confirmed this hypothesis by looking into the
> dataplane
> >> >> > >> > > > > results. Out of the 150 VMs, 10% of them were
> unreachable via ping
> >> >> > >> > > > > when using ovn-controller from master.
> >> >> > >> > > > >
> >> >> > >> > > > > @Han, others, do you have any ideas as of what could be
> happening
> >> >> > >> > > > > here? We'll be able to use this setup for a few more
> days so let me
> >> >> > >> > > > > know if you want us to pull some other data/traces, .
