Re: [ovs-discuss] [OVN] Kolla Ansible deployment

[Michał Nasiadka]

Hi Tony,

Kolla is installing ovn from RDO (CentOS) or UCA (Ubuntu) packages - and
they are more or less stable in a specific OpenStack release - so best
would be to use template override functionality and build your own images
with newer versions.

For 1. and 2. - I will submit patches to change behaviour in about two
weeks (I just started vacation).

Thanks for your valuable feedback
Michal

On Sat, 8 Aug 2020 at 01:37, Tony Liu  wrote:

> Hi Michal,
>
> Based on some comments from the community, I'd like to have your
> attention on couple things here.
>
> 1. For now, "--db-nb-create-insecure-remote=yes" is set in
> ovn-nb-db.json.j2, when deploy ovn-nb-db and ovn-sb-db.
>
> That results a TCP method is set by ovn-ctl script to run
> ovsdb-server.
>
> It's recommended to not set TCP method in command line, instead,
> create connection table for TCP method. That way, the inactivity
> probe interval can be set in the connection table as well.
>
> 2. On chassis node (compute node or gateway node), ovsdb-server
> opens TCP socket and ovn-controller connects to it. Since this
> is local communication, it's recommended to use UNIX socket.
> It also avoids inactivity probe.
>
> How can I get update on Kolla container image updates?
> For example, I am currently using ussuri, where OVN is 20.03.
> I wonder if I can get upgraded ussuri image with OVN 20.06?
> Or Kolla container upgrade is aligned with OpenStack release,
> like I won't be able to get upgraded OVN until next OpenStack
> release? If that's the case, I will just upgrade the container
> by myself.
>
>
> Thanks!
>
> Tony
> > -Original Message-
> > From: Michał Nasiadka 
> > Sent: Thursday, July 23, 2020 3:27 AM
> > To: Tony Liu 
> > Cc: Daniel Alvarez ; ovs-discuss@openvswitch.org
> > Subject: Re: [ovs-discuss] OVN scale
> >
> > Hi Tony,
> >
> > I’m the core reviewer/developer behind OVN implementation in Kolla-
> > Ansible.
> > Just to make some clarifications to the thread - yes Kolla-Ansible
> > deploys a raft cluster.
> >
> > I would be happy to see some results/reports from that OVN scale test -
> > and if you would see any improvements/bugs we could resolve - please
> > just let me know.
> >
> > Best regards,
> >
> > Michal
>
> --
Michał Nasiadka
mnasia...@gmail.com
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] [OVN] Kolla Ansible deployment

[Tony Liu]

Hi Michal,

Based on some comments from the community, I'd like to have your
attention on couple things here.

1. For now, "--db-nb-create-insecure-remote=yes" is set in
ovn-nb-db.json.j2, when deploy ovn-nb-db and ovn-sb-db.

That results a TCP method is set by ovn-ctl script to run
ovsdb-server.

It's recommended to not set TCP method in command line, instead,
create connection table for TCP method. That way, the inactivity
probe interval can be set in the connection table as well.

2. On chassis node (compute node or gateway node), ovsdb-server
opens TCP socket and ovn-controller connects to it. Since this
is local communication, it's recommended to use UNIX socket.
It also avoids inactivity probe.

How can I get update on Kolla container image updates?
For example, I am currently using ussuri, where OVN is 20.03.
I wonder if I can get upgraded ussuri image with OVN 20.06?
Or Kolla container upgrade is aligned with OpenStack release,
like I won't be able to get upgraded OVN until next OpenStack
release? If that's the case, I will just upgrade the container
by myself.


Thanks!

Tony
> -Original Message-
> From: Michał Nasiadka 
> Sent: Thursday, July 23, 2020 3:27 AM
> To: Tony Liu 
> Cc: Daniel Alvarez ; ovs-discuss@openvswitch.org
> Subject: Re: [ovs-discuss] OVN scale
> 
> Hi Tony,
> 
> I’m the core reviewer/developer behind OVN implementation in Kolla-
> Ansible.
> Just to make some clarifications to the thread - yes Kolla-Ansible
> deploys a raft cluster.
> 
> I would be happy to see some results/reports from that OVN scale test -
> and if you would see any improvements/bugs we could resolve - please
> just let me know.
> 
> Best regards,
> 
> Michal

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Double free in recent kernels after memleak fix

[Johan Knöös via discuss]

On Fri, Aug 7, 2020 at 3:20 PM Paul E. McKenney  wrote:
>
> On Fri, Aug 07, 2020 at 04:47:56PM -0400, Joel Fernandes wrote:
> > Hi,
> > Adding more of us working on RCU as well. Johan from another team at
> > Google discovered a likely issue in openswitch, details below:
> >
> > On Fri, Aug 7, 2020 at 11:32 AM Johan Knöös  wrote:
> > >
> > > On Tue, Aug 4, 2020 at 8:52 AM Gregory Rose  wrote:
> > > >
> > > >
> > > >
> > > > On 8/3/2020 12:01 PM, Johan Knöös via discuss wrote:
> > > > > Hi Open vSwitch contributors,
> > > > >
> > > > > We have found openvswitch is causing double-freeing of memory. The
> > > > > issue was not present in kernel version 5.5.17 but is present in
> > > > > 5.6.14 and newer kernels.
> > > > >
> > > > > After reverting the RCU commits below for debugging, enabling
> > > > > slub_debug, lockdep, and KASAN, we see the warnings at the end of this
> > > > > email in the kernel log (the last one shows the double-free). When I
> > > > > revert 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 ("net: openvswitch:
> > > > > fix possible memleak on destroy flow-table"), the symptoms disappear.
> > > > > While I have a reliable way to reproduce the issue, I unfortunately
> > > > > don't yet have a process that's amenable to sharing. Please take a
> > > > > look.
> > > > >
> > > > > 189a6883dcf7 rcu: Remove kfree_call_rcu_nobatch()
> > > > > 77a40f97030b rcu: Remove kfree_rcu() special casing and lazy-callback 
> > > > > handling
> > > > > e99637becb2e rcu: Add support for debug_objects debugging for 
> > > > > kfree_rcu()
> > > > > 0392bebebf26 rcu: Add multiple in-flight batches of kfree_rcu() work
> > > > > 569d767087ef rcu: Make kfree_rcu() use a non-atomic ->monitor_todo
> > > > > a35d16905efc rcu: Add basic support for kfree_rcu() batching
> >
> > Note that these reverts were only for testing the same code, because
> > he was testing 2 different kernel versions. One of them did not have
> > this set. So I asked him to revert. There's no known bug in the
> > reverted code itself. But somehow these patches do make it harder for
> > him to reproduce the issue.

I'm not certain the frequency of the issue changes with and without
these commits on 5.6.14, but at least the symptoms/definition of the
issue changes. To clarify, this is what I've observed with different
kernels:
* 5.6.14:  "kernel BUG at mm/slub.c:304!". Easily reproducible.
* 5.6.14 with the above RCU commits reverted: the warnings reported in
my original email. Easily reproducible.
* 5.6.14 with the above RCU commits reverted and
50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 reverted: no warnings
observed (the frequency might be the same as on 5.5.17).
* 5.5.17: warning at kernel/rcu/tree.c#L2239. Difficult to reproduce.
Maybe a different root cause.

> Perhaps they adjust timing?
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Good to know. Yes, I am using 20.03.
Will try to upgrade.


Thanks!

Tony
> -Original Message-
> From: Han Zhou 
> Sent: Friday, August 7, 2020 3:39 PM
> To: Tony Liu 
> Cc: ovs-dev ; ovs-discuss  disc...@openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> changes in sb-db
> 
> The change in external-ids is not monitored by ovn-controller any more
> after version 20.06. Probably you are still using an older version?
> 
> 
> On Fri, Aug 7, 2020 at 3:11 PM Tony Liu   > wrote:
> 
> 
>   Is the changes made by Neutron liveness-check in SB properly
> handled
>   by inc_proc_eng?
> 
>   Given my testing scale, even the frequency is lowered by that fix,
>   ovn-controller still takes quite lot of CPU and time to compute SB.
> 
> 
>   Thanks!
> 
>   Tony
>   > -Original Message-
>   > From: Tony Liu   >
>   > Sent: Friday, August 7, 2020 1:25 PM
>   > To: Tony Liu   >; Han Zhou   >
>   > Cc: ovs-dev mailto:ovs-
> d...@openvswitch.org> >; ovs-discuss> disc...@openvswitch.org  >
>   > Subject: RE: [ovs-discuss] [OVN] ovn-controller takes 100% cpu
> while no
>   > changes in sb-db
>   >
>   > Thanks for the hint!
>   > 
>   > 2020-08-07T19:44:18.019Z|616614|jsonrpc|DBG|tcp:10.6.20.84:6642
>  :
>   > received notification, method="update3",
>   > params=[["monid","OVN_Southbound"],"5002cb22-13e5-490a-9a64-
>   > 5d48914138ca",{"Chassis":{"0390b621-152b-48a0-a3d1-
>   >
> 2973c0b823cc":{"modify":{"external_ids":["map",[["neutron:liveness_check
>   > _at","2020-08-07T19:44:07.130233+00:00"]]]]
>   > 
>   >
>   > Nailed it...
>   > https://bugs.launchpad.net/neutron/+bug/1883554
>   >
>   >
>   > Tony
>   > > -Original Message-
>   > > From: dev mailto:ovs-dev-
> boun...@openvswitch.org> > On Behalf Of Tony Liu
>   > > Sent: Friday, August 7, 2020 1:14 PM
>   > > To: Han Zhou mailto:zhou...@gmail.com> >
>   > > Cc: ovs-dev mailto:ovs-
> d...@openvswitch.org> >; ovs-discuss> > disc...@openvswitch.org  >
>   > > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] ovn-controller takes
> 100%
>   > > cpu while no changes in sb-db
>   > >
>   > >
>   > > Here is the outpu.
>   > > 
>   > > [root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t
>   > > /run/ovn/ovn-controller.6.ctl coverage/show Event coverage, avg
> rate
>   > > over last: 5 seconds, last minute, last hour,  hash=e70a83c8:
>   > > lflow_run  0.0/sec 0.083/sec
> 0.0725/sec
>   > > total: 295
>   > > miniflow_malloc0.0/sec 44356.817/sec
> 44527.3975/sec
>   > > total: 180635403
>   > > hindex_pathological0.0/sec 0.000/sec
> 0./sec
>   > > total: 7187
>   > > hindex_expand  0.0/sec 0.000/sec
> 0./sec
>   > > total: 17
>   > > hmap_pathological  0.0/sec 4.167/sec
> 4.1806/sec
>   > > total: 25091
>   > > hmap_expand0.0/sec  5366.500/sec
> 5390.0800/sec
>   > > total: 23680738
>   > > txn_unchanged  0.0/sec 0.300/sec
> 0.3175/sec
>   > > total: 11024
>   > > txn_incomplete 0.0/sec 0.100/sec
> 0.0836/sec
>   > > total: 974
>   > > txn_success0.0/sec 0.033/sec
> 0.0308/sec
>   > > total: 129
>   > > txn_try_again  0.0/sec 0.000/sec
> 0.0003/sec
>   > > total: 1
>   > > poll_create_node   0.4/sec 1.933/sec
> 1.9575/sec
>   > > total: 55611
>   > > poll_zero_timeout  0.0/sec 0.067/sec
> 0.0556/sec
>   > > total: 241
>   > > rconn_queued   0.0/sec 0.050/sec
> 0.0594/sec
>   > > total: 1208720
>   > > rconn_sent 0.0/sec 0.050/sec
> 0.0594/sec
>   > > total: 1208720
>   > > seq_change 0.2/sec 0.783/sec
> 0.7492/sec
>   > > total: 13962
>   > > pstream_open   0.0/sec 0.000/sec
> 0./sec
>   > > total: 1
>   > > stream_open0.0/sec 0.000/sec
> 0.0003/sec
>   > > total: 5
>   > > unixctl_received   0.0/sec 0.000/sec
> 0.0011/sec
>   > > total: 4
>   > > unixctl_replied0.0/sec 0.000/sec
> 0.0011/sec
>   > > total: 4
>   > > util_xalloc0.8/sec 1396586.967/sec
> 240916.6047/sec
>   > > total: 5834154064
>   > > vconn_open 0.0/sec 0.000/sec
> 0./sec
>   > > total: 2
>   > > vconn_received 0.0/sec 0.050/sec
> 0.0594/sec
>   > > total: 632
>   > > 

Re: [ovs-discuss] Double free in recent kernels after memleak fix

[Paul E. McKenney]

On Fri, Aug 07, 2020 at 04:47:56PM -0400, Joel Fernandes wrote:
> Hi,
> Adding more of us working on RCU as well. Johan from another team at
> Google discovered a likely issue in openswitch, details below:
> 
> On Fri, Aug 7, 2020 at 11:32 AM Johan Knöös  wrote:
> >
> > On Tue, Aug 4, 2020 at 8:52 AM Gregory Rose  wrote:
> > >
> > >
> > >
> > > On 8/3/2020 12:01 PM, Johan Knöös via discuss wrote:
> > > > Hi Open vSwitch contributors,
> > > >
> > > > We have found openvswitch is causing double-freeing of memory. The
> > > > issue was not present in kernel version 5.5.17 but is present in
> > > > 5.6.14 and newer kernels.
> > > >
> > > > After reverting the RCU commits below for debugging, enabling
> > > > slub_debug, lockdep, and KASAN, we see the warnings at the end of this
> > > > email in the kernel log (the last one shows the double-free). When I
> > > > revert 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 ("net: openvswitch:
> > > > fix possible memleak on destroy flow-table"), the symptoms disappear.
> > > > While I have a reliable way to reproduce the issue, I unfortunately
> > > > don't yet have a process that's amenable to sharing. Please take a
> > > > look.
> > > >
> > > > 189a6883dcf7 rcu: Remove kfree_call_rcu_nobatch()
> > > > 77a40f97030b rcu: Remove kfree_rcu() special casing and lazy-callback 
> > > > handling
> > > > e99637becb2e rcu: Add support for debug_objects debugging for 
> > > > kfree_rcu()
> > > > 0392bebebf26 rcu: Add multiple in-flight batches of kfree_rcu() work
> > > > 569d767087ef rcu: Make kfree_rcu() use a non-atomic ->monitor_todo
> > > > a35d16905efc rcu: Add basic support for kfree_rcu() batching
> 
> Note that these reverts were only for testing the same code, because
> he was testing 2 different kernel versions. One of them did not have
> this set. So I asked him to revert. There's no known bug in the
> reverted code itself. But somehow these patches do make it harder for
> him to reproduce the issue.

Perhaps they adjust timing?

> > > > Thanks,
> > > > Johan Knöös
> > >
> > > Let's add the author of the patch you reverted and the Linux netdev
> > > mailing list.
> > >
> > > - Greg
> >
> > I found we also sometimes get warnings from
> > https://elixir.bootlin.com/linux/v5.5.17/source/kernel/rcu/tree.c#L2239
> > under similar conditions even on kernel 5.5.17, which I believe may be
> > related. However, it's much rarer and I don't have a reliable way of
> > reproducing it. Perhaps 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 only
> > increases the frequency of a pre-existing bug.
> 
> This is interesting, because I saw kbuild warn me recently [1] about
> it as well. Though, I was actually intentionally messing with the
> segcblist. I plan to debug it next week, but the warning itself is
> unlikely to be caused by my patch IMHO (since it is slightly
> orthogonal to what I changed).
> 
> [1] https://lore.kernel.org/lkml/20200720005334.GC19262@shao2-debian/
> 
> But then again, I have not heard reports of this warning firing. Paul,
> has this come to your radar recently?

I have not seen any recent WARNs in rcu_do_batch().  I am guessing that
this is one of the last two in that function?

If so, have you tried using CONFIG_DEBUG_OBJECTS_RCU_HEAD=y?  That Kconfig
option is designed to help locate double frees via RCU.

Thanx, Paul
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] [OVN] log file rotation and time zone

[Tony Liu]

Hi,

Is log file rotation and compression supported for all
OVN services?

How is the time stamp time zone determined by logging?
Is it /etc/localtime, hardcoded UTC, or anything else?


Thanks!

Tony

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Is the changes made by Neutron liveness-check in SB properly handled
by inc_proc_eng?

Given my testing scale, even the frequency is lowered by that fix,
ovn-controller still takes quite lot of CPU and time to compute SB.


Thanks!

Tony
> -Original Message-
> From: Tony Liu 
> Sent: Friday, August 7, 2020 1:25 PM
> To: Tony Liu ; Han Zhou 
> Cc: ovs-dev ; ovs-discuss  disc...@openvswitch.org>
> Subject: RE: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> changes in sb-db
> 
> Thanks for the hint!
> 
> 2020-08-07T19:44:18.019Z|616614|jsonrpc|DBG|tcp:10.6.20.84:6642:
> received notification, method="update3",
> params=[["monid","OVN_Southbound"],"5002cb22-13e5-490a-9a64-
> 5d48914138ca",{"Chassis":{"0390b621-152b-48a0-a3d1-
> 2973c0b823cc":{"modify":{"external_ids":["map",[["neutron:liveness_check
> _at","2020-08-07T19:44:07.130233+00:00"]]]]
> 
> 
> Nailed it...
> https://bugs.launchpad.net/neutron/+bug/1883554
> 
> 
> Tony
> > -Original Message-
> > From: dev  On Behalf Of Tony Liu
> > Sent: Friday, August 7, 2020 1:14 PM
> > To: Han Zhou 
> > Cc: ovs-dev ; ovs-discuss  > disc...@openvswitch.org>
> > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] ovn-controller takes 100%
> > cpu while no changes in sb-db
> >
> >
> > Here is the outpu.
> > 
> > [root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t
> > /run/ovn/ovn-controller.6.ctl coverage/show Event coverage, avg rate
> > over last: 5 seconds, last minute, last hour,  hash=e70a83c8:
> > lflow_run  0.0/sec 0.083/sec0.0725/sec
> > total: 295
> > miniflow_malloc0.0/sec 44356.817/sec44527.3975/sec
> > total: 180635403
> > hindex_pathological0.0/sec 0.000/sec0./sec
> > total: 7187
> > hindex_expand  0.0/sec 0.000/sec0./sec
> > total: 17
> > hmap_pathological  0.0/sec 4.167/sec4.1806/sec
> > total: 25091
> > hmap_expand0.0/sec  5366.500/sec 5390.0800/sec
> > total: 23680738
> > txn_unchanged  0.0/sec 0.300/sec0.3175/sec
> > total: 11024
> > txn_incomplete 0.0/sec 0.100/sec0.0836/sec
> > total: 974
> > txn_success0.0/sec 0.033/sec0.0308/sec
> > total: 129
> > txn_try_again  0.0/sec 0.000/sec0.0003/sec
> > total: 1
> > poll_create_node   0.4/sec 1.933/sec1.9575/sec
> > total: 55611
> > poll_zero_timeout  0.0/sec 0.067/sec0.0556/sec
> > total: 241
> > rconn_queued   0.0/sec 0.050/sec0.0594/sec
> > total: 1208720
> > rconn_sent 0.0/sec 0.050/sec0.0594/sec
> > total: 1208720
> > seq_change 0.2/sec 0.783/sec0.7492/sec
> > total: 13962
> > pstream_open   0.0/sec 0.000/sec0./sec
> > total: 1
> > stream_open0.0/sec 0.000/sec0.0003/sec
> > total: 5
> > unixctl_received   0.0/sec 0.000/sec0.0011/sec
> > total: 4
> > unixctl_replied0.0/sec 0.000/sec0.0011/sec
> > total: 4
> > util_xalloc0.8/sec 1396586.967/sec   240916.6047/sec
> > total: 5834154064
> > vconn_open 0.0/sec 0.000/sec0./sec
> > total: 2
> > vconn_received 0.0/sec 0.050/sec0.0594/sec
> > total: 632
> > vconn_sent 0.0/sec 0.050/sec0.0494/sec
> > total: 1213248
> > netlink_received   0.0/sec 0.300/sec0.2900/sec
> > total: 1188
> > netlink_recv_jumbo 0.0/sec 0.083/sec0.0725/sec
> > total: 296
> > netlink_sent   0.0/sec 0.300/sec0.2900/sec
> > total: 1188
> > cmap_expand0.0/sec 0.000/sec0./sec
> > total: 3
> > 82 events never hit
> > [root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t
> > /run/ovn/ovn-controller.6.ctl coverage/show Event coverage, avg rate
> > over last: 5 seconds, last minute, last hour,  hash=d0107601:
> > lflow_run  0.2/sec 0.083/sec0.0717/sec
> > total: 296
> > miniflow_malloc  122834.2/sec 51180.917/sec43930.2869/sec
> > total: 181249574
> > hindex_pathological0.0/sec 0.000/sec0./sec
> > total: 7187
> > hindex_expand  0.0/sec 0.000/sec0./sec
> > total: 17
> > hmap_pathological 13.2/sec 4.967/sec4.1264/sec
> > total: 25157
> > hmap_expand  14982.2/sec  6205.067/sec 5317.9547/sec
> > total: 23755649
> > txn_unchanged  1.4/sec 0.400/sec0.3144/sec
> > total: 11031
> > txn_incomplete 0.4/sec 0.117/sec0.0825/sec
> > total: 976
> > txn_success0.2/sec 0.050/sec0.0306/sec
> > total: 130
> > txn_try_again  0.0/sec 0.000/sec0.0003/sec
> > total: 1
> > 

Re: [ovs-discuss] Double free in recent kernels after memleak fix

[Cong Wang]

On Fri, Aug 7, 2020 at 8:33 AM Johan Knöös  wrote:
>
> On Tue, Aug 4, 2020 at 8:52 AM Gregory Rose  wrote:
> >
> >
> >
> > On 8/3/2020 12:01 PM, Johan Knöös via discuss wrote:
> > > Hi Open vSwitch contributors,
> > >
> > > We have found openvswitch is causing double-freeing of memory. The
> > > issue was not present in kernel version 5.5.17 but is present in
> > > 5.6.14 and newer kernels.
> > >
> > > After reverting the RCU commits below for debugging, enabling
> > > slub_debug, lockdep, and KASAN, we see the warnings at the end of this
> > > email in the kernel log (the last one shows the double-free). When I
> > > revert 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 ("net: openvswitch:
> > > fix possible memleak on destroy flow-table"), the symptoms disappear.
> > > While I have a reliable way to reproduce the issue, I unfortunately
> > > don't yet have a process that's amenable to sharing. Please take a
> > > look.
> > >
> > > 189a6883dcf7 rcu: Remove kfree_call_rcu_nobatch()
> > > 77a40f97030b rcu: Remove kfree_rcu() special casing and lazy-callback 
> > > handling
> > > e99637becb2e rcu: Add support for debug_objects debugging for kfree_rcu()
> > > 0392bebebf26 rcu: Add multiple in-flight batches of kfree_rcu() work
> > > 569d767087ef rcu: Make kfree_rcu() use a non-atomic ->monitor_todo
> > > a35d16905efc rcu: Add basic support for kfree_rcu() batching
> > >
> > > Thanks,
> > > Johan Knöös
> >
> > Let's add the author of the patch you reverted and the Linux netdev
> > mailing list.
> >
> > - Greg
>
> I found we also sometimes get warnings from
> https://elixir.bootlin.com/linux/v5.5.17/source/kernel/rcu/tree.c#L2239
> under similar conditions even on kernel 5.5.17, which I believe may be
> related. However, it's much rarer and I don't have a reliable way of
> reproducing it. Perhaps 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 only
> increases the frequency of a pre-existing bug.

It seems clear we have a double free on table->mask_array when
the reallocation is triggered on the destroy path.

Are you able to test the attached patch (compile tested only)?
Also note: it is generated against the latest net tree, it may not be
applied cleanly to any earlier stable release.

Thanks!
diff --git a/net/openvswitch/flow_table.c b/net/openvswitch/flow_table.c
index 8c12675cbb67..cc7859db445a 100644
--- a/net/openvswitch/flow_table.c
+++ b/net/openvswitch/flow_table.c
@@ -294,7 +294,7 @@ static int tbl_mask_array_add_mask(struct flow_table *tbl,
 }
 
 static void tbl_mask_array_del_mask(struct flow_table *tbl,
-struct sw_flow_mask *mask)
+struct sw_flow_mask *mask, bool destroy)
 {
 	struct mask_array *ma = ovsl_dereference(tbl->mask_array);
 	int i, ma_count = READ_ONCE(ma->count);
@@ -314,6 +314,11 @@ static void tbl_mask_array_del_mask(struct flow_table *tbl,
 	rcu_assign_pointer(ma->masks[i], ma->masks[ma_count -1]);
 	RCU_INIT_POINTER(ma->masks[ma_count -1], NULL);
 
+	if (destroy) {
+		kfree(mask);
+		return;
+	}
+
 	kfree_rcu(mask, rcu);
 
 	/* Shrink the mask array if necessary. */
@@ -326,7 +331,8 @@ static void tbl_mask_array_del_mask(struct flow_table *tbl,
 }
 
 /* Remove 'mask' from the mask list, if it is not needed any more. */
-static void flow_mask_remove(struct flow_table *tbl, struct sw_flow_mask *mask)
+static void flow_mask_remove(struct flow_table *tbl, struct sw_flow_mask *mask,
+			 bool destroy)
 {
 	if (mask) {
 		/* ovs-lock is required to protect mask-refcount and
@@ -337,7 +343,7 @@ static void flow_mask_remove(struct flow_table *tbl, struct sw_flow_mask *mask)
 		mask->ref_count--;
 
 		if (!mask->ref_count)
-			tbl_mask_array_del_mask(tbl, mask);
+			tbl_mask_array_del_mask(tbl, mask, destroy);
 	}
 }
 
@@ -470,7 +476,7 @@ static void table_instance_flow_free(struct flow_table *table,
 			table->ufid_count--;
 	}
 
-	flow_mask_remove(table, flow->mask);
+	flow_mask_remove(table, flow->mask, !count);
 }
 
 static void table_instance_destroy(struct flow_table *table,
@@ -521,9 +527,9 @@ void ovs_flow_tbl_destroy(struct flow_table *table)
 	struct mask_cache *mc = rcu_dereference_raw(table->mask_cache);
 	struct mask_array *ma = rcu_dereference_raw(table->mask_array);
 
+	table_instance_destroy(table, ti, ufid_ti, false);
 	call_rcu(>rcu, mask_cache_rcu_cb);
 	call_rcu(>rcu, mask_array_rcu_cb);
-	table_instance_destroy(table, ti, ufid_ti, false);
 }
 
 struct sw_flow *ovs_flow_tbl_dump_next(struct table_instance *ti,
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Han Zhou]

On Fri, Aug 7, 2020 at 1:56 PM Venugopal Iyer  wrote:

> Hi, Han:
>
>
>
> An additional comment;
>
>
>
> *From:* ovn-kuberne...@googlegroups.com  *On
> Behalf Of *Venugopal Iyer
> *Sent:* Friday, August 7, 2020 1:51 PM
> *To:* Han Zhou ; Numan Siddique 
> *Cc:* Winson Wang ; ovs-discuss@openvswitch.org;
> ovn-kuberne...@googlegroups.com; Dumitru Ceara ; Han
> Zhou 
> *Subject:* RE: ovn-k8s scale: how to make new ovn-controller process keep
> the previous Open Flow in br-int
>
>
>
> *External email: Use caution opening links or attachments*
>
>
>
> Hi, Han:
>
>
>
> *From:* ovn-kuberne...@googlegroups.com  *On
> Behalf Of *Han Zhou
> *Sent:* Friday, August 7, 2020 1:04 PM
> *To:* Numan Siddique 
> *Cc:* Venugopal Iyer ; Winson Wang <
> windson.w...@gmail.com>; ovs-discuss@openvswitch.org;
> ovn-kuberne...@googlegroups.com; Dumitru Ceara ; Han
> Zhou 
> *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process keep
> the previous Open Flow in br-int
>
>
>
> *External email: Use caution opening links or attachments*
>
>
>
>
>
>
>
> On Fri, Aug 7, 2020 at 12:35 PM Numan Siddique  wrote:
>
>
>
>
>
> On Sat, Aug 8, 2020 at 12:16 AM Han Zhou  wrote:
>
>
>
>
>
> On Thu, Aug 6, 2020 at 10:22 AM Han Zhou  wrote:
>
>
>
>
>
> On Thu, Aug 6, 2020 at 9:15 AM Numan Siddique  wrote:
>
>
>
>
>
> On Thu, Aug 6, 2020 at 9:25 PM Venugopal Iyer 
> wrote:
>
> Hi, Han:
>
>
>
> A comment inline:
>
>
>
> *From:* ovn-kuberne...@googlegroups.com  *On
> Behalf Of *Han Zhou
> *Sent:* Wednesday, August 5, 2020 3:36 PM
> *To:* Winson Wang 
> *Cc:* ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com;
> Dumitru Ceara ; Han Zhou 
> *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process keep
> the previous Open Flow in br-int
>
>
>
> *External email: Use caution opening links or attachments*
>
>
>
>
>
>
>
> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
> wrote:
>
> Hello OVN Experts,
>
>
> With ovn-k8s,  we need to keep the flows always on br-int which needed by
> running pods on the k8s node.
>
> Is there an ongoing project to address this problem?
>
> If not,  I have one proposal not sure if it is doable.
>
> Please share your thoughts.
> The issue:
>
> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
> every K8s node.  When we restart ovn-controller for upgrade using
> `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
> works fine since br-int with flows still be Installed.
>
>
>
> However, when a new ovn-controller starts it will connect OVS IDL and do
> an engine init run,  clearing all OpenFlow flows and install flows based on
> SB DB.
>
> With open flows count above 200K+,  it took more than 15 seconds to get
> all the flows installed br-int bridge again.
>
>
> Proposal solution for the issue:
>
> When the ovn-controller gets “exit --start”,  it will write a
> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
> external-ids column. When new ovn-controller starts, it will check if the
> “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
> OVS IDL to decide if it will force a recomputing process?
>
>
>
>
>
> Hi Winson,
>
>
>
> Thanks for the proposal. Yes, the connection break during upgrading is a
> real issue in a large scale environment. However, the proposal doesn't
> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
> which is a completely different connection from the ovs-vswitchd open-flow
> connection.
>
> To avoid clearing the open-flow table during ovn-controller startup, we
> can find a way to postpone clearing the OVS flows after the recomputing in
> ovn-controller is completed, right before ovn-controller replacing with the
> new flows.
>
> *[vi> ] *
>
> *[vi> ] Seems like we force recompute today if the OVS IDL is reconnected.
> Would it be possible to defer *
>
> *decision to  recompute the flows based on  the  SB’s nb_cfg we have
>  sync’d with? i.e.  If  our nb_cfg is *
>
> *in sync with the SB’s global nb_cfg, we can skip the recompute?  At least
> if nothing has changed since*
>
> *the restart, we won’t need to do anything.. We could stash nb_cfg in OVS
> (once ovn-controller receives*
>
> *conformation from OVS that the physical flows for an nb_cfg update are in
> place), which should be cleared if *
>
> *OVS itself is restarted.. (I mean currently, nb_cfg is used to check if
> NB, SB and Chassis are in sync, we *
>
> *could extend this to OVS/physical flows?)*
>
>
>
> *Have not thought through this though .. so maybe I am missing something…*
>
>
>
> *Thanks,*
>
>
>
> *-venu*
>
> This should largely reduce the time of connection broken during upgrading.
> Some changes in the ofctrl module's state machine are required, but I am
> not 100% sure if this approach is applicable. Need to check more details.
>
>
>
>
>
> We can also think if its possible to do the below way
>
>- When ovn-controller starts, it will not clear the flows, but instead
> will get the dump of 

Re: [ovs-discuss] Double free in recent kernels after memleak fix

[Joel Fernandes]

Hi,
Adding more of us working on RCU as well. Johan from another team at
Google discovered a likely issue in openswitch, details below:

On Fri, Aug 7, 2020 at 11:32 AM Johan Knöös  wrote:
>
> On Tue, Aug 4, 2020 at 8:52 AM Gregory Rose  wrote:
> >
> >
> >
> > On 8/3/2020 12:01 PM, Johan Knöös via discuss wrote:
> > > Hi Open vSwitch contributors,
> > >
> > > We have found openvswitch is causing double-freeing of memory. The
> > > issue was not present in kernel version 5.5.17 but is present in
> > > 5.6.14 and newer kernels.
> > >
> > > After reverting the RCU commits below for debugging, enabling
> > > slub_debug, lockdep, and KASAN, we see the warnings at the end of this
> > > email in the kernel log (the last one shows the double-free). When I
> > > revert 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 ("net: openvswitch:
> > > fix possible memleak on destroy flow-table"), the symptoms disappear.
> > > While I have a reliable way to reproduce the issue, I unfortunately
> > > don't yet have a process that's amenable to sharing. Please take a
> > > look.
> > >
> > > 189a6883dcf7 rcu: Remove kfree_call_rcu_nobatch()
> > > 77a40f97030b rcu: Remove kfree_rcu() special casing and lazy-callback 
> > > handling
> > > e99637becb2e rcu: Add support for debug_objects debugging for kfree_rcu()
> > > 0392bebebf26 rcu: Add multiple in-flight batches of kfree_rcu() work
> > > 569d767087ef rcu: Make kfree_rcu() use a non-atomic ->monitor_todo
> > > a35d16905efc rcu: Add basic support for kfree_rcu() batching
> > >

Note that these reverts were only for testing the same code, because
he was testing 2 different kernel versions. One of them did not have
this set. So I asked him to revert. There's no known bug in the
reverted code itself. But somehow these patches do make it harder for
him to reproduce the issue.

> > > Thanks,
> > > Johan Knöös
> >
> > Let's add the author of the patch you reverted and the Linux netdev
> > mailing list.
> >
> > - Greg
>
> I found we also sometimes get warnings from
> https://elixir.bootlin.com/linux/v5.5.17/source/kernel/rcu/tree.c#L2239
> under similar conditions even on kernel 5.5.17, which I believe may be
> related. However, it's much rarer and I don't have a reliable way of
> reproducing it. Perhaps 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 only
> increases the frequency of a pre-existing bug.

This is interesting, because I saw kbuild warn me recently [1] about
it as well. Though, I was actually intentionally messing with the
segcblist. I plan to debug it next week, but the warning itself is
unlikely to be caused by my patch IMHO (since it is slightly
orthogonal to what I changed).

[1] https://lore.kernel.org/lkml/20200720005334.GC19262@shao2-debian/

But then again, I have not heard reports of this warning firing. Paul,
has this come to your radar recently?

Thanks,

 - Joel
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Double free in recent kernels after memleak fix

[Joel Fernandes]

On Fri, Aug 7, 2020 at 4:47 PM Joel Fernandes  wrote:
>
> Hi,
> Adding more of us working on RCU as well. Johan from another team at
> Google discovered a likely issue in openswitch, details below:
>
> On Fri, Aug 7, 2020 at 11:32 AM Johan Knöös  wrote:
> >
> > On Tue, Aug 4, 2020 at 8:52 AM Gregory Rose  wrote:
> > >
> > >
> > >
> > > On 8/3/2020 12:01 PM, Johan Knöös via discuss wrote:
> > > > Hi Open vSwitch contributors,
> > > >
> > > > We have found openvswitch is causing double-freeing of memory. The
> > > > issue was not present in kernel version 5.5.17 but is present in
> > > > 5.6.14 and newer kernels.
> > > >
> > > > After reverting the RCU commits below for debugging, enabling
> > > > slub_debug, lockdep, and KASAN, we see the warnings at the end of this
> > > > email in the kernel log (the last one shows the double-free). When I
> > > > revert 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 ("net: openvswitch:
> > > > fix possible memleak on destroy flow-table"), the symptoms disappear.
> > > > While I have a reliable way to reproduce the issue, I unfortunately
> > > > don't yet have a process that's amenable to sharing. Please take a
> > > > look.
> > > >
> > > > 189a6883dcf7 rcu: Remove kfree_call_rcu_nobatch()
> > > > 77a40f97030b rcu: Remove kfree_rcu() special casing and lazy-callback 
> > > > handling
> > > > e99637becb2e rcu: Add support for debug_objects debugging for 
> > > > kfree_rcu()
> > > > 0392bebebf26 rcu: Add multiple in-flight batches of kfree_rcu() work
> > > > 569d767087ef rcu: Make kfree_rcu() use a non-atomic ->monitor_todo
> > > > a35d16905efc rcu: Add basic support for kfree_rcu() batching
> > > >
>
> Note that these reverts were only for testing the same code, because
> he was testing 2 different kernel versions. One of them did not have
> this set. So I asked him to revert. There's no known bug in the
> reverted code itself. But somehow these patches do make it harder for
> him to reproduce the issue.

And the reason for this is likely the additional kfree batching is
slowing down the occurrence of the crash.

thanks,

 - Joel
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Venugopal Iyer]

Hi, Han:

An additional comment;

From: ovn-kuberne...@googlegroups.com  On 
Behalf Of Venugopal Iyer
Sent: Friday, August 7, 2020 1:51 PM
To: Han Zhou ; Numan Siddique 
Cc: Winson Wang ; ovs-discuss@openvswitch.org; 
ovn-kuberne...@googlegroups.com; Dumitru Ceara ; Han Zhou 

Subject: RE: ovn-k8s scale: how to make new ovn-controller process keep the 
previous Open Flow in br-int

External email: Use caution opening links or attachments

Hi, Han:

From: ovn-kuberne...@googlegroups.com 
mailto:ovn-kuberne...@googlegroups.com>> On 
Behalf Of Han Zhou
Sent: Friday, August 7, 2020 1:04 PM
To: Numan Siddique mailto:num...@ovn.org>>
Cc: Venugopal Iyer mailto:venugop...@nvidia.com>>; 
Winson Wang mailto:windson.w...@gmail.com>>; 
ovs-discuss@openvswitch.org; 
ovn-kuberne...@googlegroups.com; 
Dumitru Ceara mailto:dce...@redhat.com>>; Han Zhou 
mailto:hz...@ovn.org>>
Subject: Re: ovn-k8s scale: how to make new ovn-controller process keep the 
previous Open Flow in br-int

External email: Use caution opening links or attachments



On Fri, Aug 7, 2020 at 12:35 PM Numan Siddique 
mailto:num...@ovn.org>> wrote:


On Sat, Aug 8, 2020 at 12:16 AM Han Zhou 
mailto:zhou...@gmail.com>> wrote:


On Thu, Aug 6, 2020 at 10:22 AM Han Zhou 
mailto:zhou...@gmail.com>> wrote:


On Thu, Aug 6, 2020 at 9:15 AM Numan Siddique 
mailto:num...@ovn.org>> wrote:


On Thu, Aug 6, 2020 at 9:25 PM Venugopal Iyer 
mailto:venugop...@nvidia.com>> wrote:
Hi, Han:

A comment inline:

From: ovn-kuberne...@googlegroups.com 
mailto:ovn-kuberne...@googlegroups.com>> On 
Behalf Of Han Zhou
Sent: Wednesday, August 5, 2020 3:36 PM
To: Winson Wang mailto:windson.w...@gmail.com>>
Cc: ovs-discuss@openvswitch.org; 
ovn-kuberne...@googlegroups.com; 
Dumitru Ceara mailto:dce...@redhat.com>>; Han Zhou 
mailto:hz...@ovn.org>>
Subject: Re: ovn-k8s scale: how to make new ovn-controller process keep the 
previous Open Flow in br-int

External email: Use caution opening links or attachments



On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
mailto:windson.w...@gmail.com>> wrote:
Hello OVN Experts,

With ovn-k8s,  we need to keep the flows always on br-int which needed by 
running pods on the k8s node.
Is there an ongoing project to address this problem?
If not,  I have one proposal not sure if it is doable.
Please share your thoughts.
The issue:

In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on every 
K8s node.  When we restart ovn-controller for upgrade using `ovs-appctl -t 
ovn-controller exit --restart`,  the remaining traffic still works fine since 
br-int with flows still be Installed.


However, when a new ovn-controller starts it will connect OVS IDL and do an 
engine init run,  clearing all OpenFlow flows and install flows based on SB DB.

With open flows count above 200K+,  it took more than 15 seconds to get all the 
flows installed br-int bridge again.

Proposal solution for the issue:

When the ovn-controller gets “exit --start”,  it will write a  “ovs-cond-seqno” 
to OVS IDL and store the value to Open vSwitch table in external-ids column. 
When new ovn-controller starts, it will check if the “ovs-cond-seqno” exists in 
the Open_vSwitch table,  and get the seqno from OVS IDL to decide if it will 
force a recomputing process?


Hi Winson,

Thanks for the proposal. Yes, the connection break during upgrading is a real 
issue in a large scale environment. However, the proposal doesn't work. The 
"ovs-cond-seqno" is for the OVSDB IDL for the local conf DB, which is a 
completely different connection from the ovs-vswitchd open-flow connection.
To avoid clearing the open-flow table during ovn-controller startup, we can 
find a way to postpone clearing the OVS flows after the recomputing in 
ovn-controller is completed, right before ovn-controller replacing with the new 
flows.
[vi> ]
[vi> ] Seems like we force recompute today if the OVS IDL is reconnected. Would 
it be possible to defer
decision to  recompute the flows based on  the  SB’s nb_cfg we have  sync’d 
with? i.e.  If  our nb_cfg is
in sync with the SB’s global nb_cfg, we can skip the recompute?  At least if 
nothing has changed since
the restart, we won’t need to do anything.. We could stash nb_cfg in OVS (once 
ovn-controller receives
conformation from OVS that the physical flows for an nb_cfg update are in 
place), which should be cleared if
OVS itself is restarted.. (I mean currently, nb_cfg is used to check if NB, SB 
and Chassis are in sync, we
could extend this to OVS/physical flows?)

Have not thought through this though .. so maybe I am missing something…

Thanks,

-venu
This should largely reduce the time of connection broken during upgrading. Some 
changes in the ofctrl module's state machine are required, but I am not 100% 
sure if this 

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Venugopal Iyer]

Hi, Han:

From: ovn-kuberne...@googlegroups.com  On 
Behalf Of Han Zhou
Sent: Friday, August 7, 2020 1:04 PM
To: Numan Siddique 
Cc: Venugopal Iyer ; Winson Wang 
; ovs-discuss@openvswitch.org; 
ovn-kuberne...@googlegroups.com; Dumitru Ceara ; Han Zhou 

Subject: Re: ovn-k8s scale: how to make new ovn-controller process keep the 
previous Open Flow in br-int

External email: Use caution opening links or attachments



On Fri, Aug 7, 2020 at 12:35 PM Numan Siddique 
mailto:num...@ovn.org>> wrote:


On Sat, Aug 8, 2020 at 12:16 AM Han Zhou 
mailto:zhou...@gmail.com>> wrote:


On Thu, Aug 6, 2020 at 10:22 AM Han Zhou 
mailto:zhou...@gmail.com>> wrote:


On Thu, Aug 6, 2020 at 9:15 AM Numan Siddique 
mailto:num...@ovn.org>> wrote:


On Thu, Aug 6, 2020 at 9:25 PM Venugopal Iyer 
mailto:venugop...@nvidia.com>> wrote:
Hi, Han:

A comment inline:

From: ovn-kuberne...@googlegroups.com 
mailto:ovn-kuberne...@googlegroups.com>> On 
Behalf Of Han Zhou
Sent: Wednesday, August 5, 2020 3:36 PM
To: Winson Wang mailto:windson.w...@gmail.com>>
Cc: ovs-discuss@openvswitch.org; 
ovn-kuberne...@googlegroups.com; 
Dumitru Ceara mailto:dce...@redhat.com>>; Han Zhou 
mailto:hz...@ovn.org>>
Subject: Re: ovn-k8s scale: how to make new ovn-controller process keep the 
previous Open Flow in br-int

External email: Use caution opening links or attachments



On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
mailto:windson.w...@gmail.com>> wrote:
Hello OVN Experts,

With ovn-k8s,  we need to keep the flows always on br-int which needed by 
running pods on the k8s node.
Is there an ongoing project to address this problem?
If not,  I have one proposal not sure if it is doable.
Please share your thoughts.
The issue:

In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on every 
K8s node.  When we restart ovn-controller for upgrade using `ovs-appctl -t 
ovn-controller exit --restart`,  the remaining traffic still works fine since 
br-int with flows still be Installed.


However, when a new ovn-controller starts it will connect OVS IDL and do an 
engine init run,  clearing all OpenFlow flows and install flows based on SB DB.

With open flows count above 200K+,  it took more than 15 seconds to get all the 
flows installed br-int bridge again.

Proposal solution for the issue:

When the ovn-controller gets “exit --start”,  it will write a  “ovs-cond-seqno” 
to OVS IDL and store the value to Open vSwitch table in external-ids column. 
When new ovn-controller starts, it will check if the “ovs-cond-seqno” exists in 
the Open_vSwitch table,  and get the seqno from OVS IDL to decide if it will 
force a recomputing process?


Hi Winson,

Thanks for the proposal. Yes, the connection break during upgrading is a real 
issue in a large scale environment. However, the proposal doesn't work. The 
"ovs-cond-seqno" is for the OVSDB IDL for the local conf DB, which is a 
completely different connection from the ovs-vswitchd open-flow connection.
To avoid clearing the open-flow table during ovn-controller startup, we can 
find a way to postpone clearing the OVS flows after the recomputing in 
ovn-controller is completed, right before ovn-controller replacing with the new 
flows.
[vi> ]
[vi> ] Seems like we force recompute today if the OVS IDL is reconnected. Would 
it be possible to defer
decision to  recompute the flows based on  the  SB’s nb_cfg we have  sync’d 
with? i.e.  If  our nb_cfg is
in sync with the SB’s global nb_cfg, we can skip the recompute?  At least if 
nothing has changed since
the restart, we won’t need to do anything.. We could stash nb_cfg in OVS (once 
ovn-controller receives
conformation from OVS that the physical flows for an nb_cfg update are in 
place), which should be cleared if
OVS itself is restarted.. (I mean currently, nb_cfg is used to check if NB, SB 
and Chassis are in sync, we
could extend this to OVS/physical flows?)

Have not thought through this though .. so maybe I am missing something…

Thanks,

-venu
This should largely reduce the time of connection broken during upgrading. Some 
changes in the ofctrl module's state machine are required, but I am not 100% 
sure if this approach is applicable. Need to check more details.


We can also think if its possible to do the below way
   - When ovn-controller starts, it will not clear the flows, but instead will 
get the dump of flows  from the br-int and populate these flows in its 
installed flows
- And then when it connects to the SB DB and computes the desired flows, it 
will anyway sync up with the installed flows with the desired flows
- And if there is no difference between desired flows and installed flows, 
there will be no impact on the datapath at all.

Although this would require a careful thought and proper handling.

Numan, as I responded to Girish, this avoids the time spent on the one-time 
flow installation after restart (the 

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Thanks for the hint!

2020-08-07T19:44:18.019Z|616614|jsonrpc|DBG|tcp:10.6.20.84:6642: received 
notification, method="update3", 
params=[["monid","OVN_Southbound"],"5002cb22-13e5-490a-9a64-5d48914138ca",{"Chassis":{"0390b621-152b-48a0-a3d1-2973c0b823cc":{"modify":{"external_ids":["map",[["neutron:liveness_check_at","2020-08-07T19:44:07.130233+00:00"]]]]


Nailed it...
https://bugs.launchpad.net/neutron/+bug/1883554


Tony
> -Original Message-
> From: dev  On Behalf Of Tony Liu
> Sent: Friday, August 7, 2020 1:14 PM
> To: Han Zhou 
> Cc: ovs-dev ; ovs-discuss  disc...@openvswitch.org>
> Subject: Re: [ovs-dev] [ovs-discuss] [OVN] ovn-controller takes 100% cpu
> while no changes in sb-db
> 
> 
> Here is the outpu.
> 
> [root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t
> /run/ovn/ovn-controller.6.ctl coverage/show Event coverage, avg rate
> over last: 5 seconds, last minute, last hour,  hash=e70a83c8:
> lflow_run  0.0/sec 0.083/sec0.0725/sec
> total: 295
> miniflow_malloc0.0/sec 44356.817/sec44527.3975/sec
> total: 180635403
> hindex_pathological0.0/sec 0.000/sec0./sec
> total: 7187
> hindex_expand  0.0/sec 0.000/sec0./sec
> total: 17
> hmap_pathological  0.0/sec 4.167/sec4.1806/sec
> total: 25091
> hmap_expand0.0/sec  5366.500/sec 5390.0800/sec
> total: 23680738
> txn_unchanged  0.0/sec 0.300/sec0.3175/sec
> total: 11024
> txn_incomplete 0.0/sec 0.100/sec0.0836/sec
> total: 974
> txn_success0.0/sec 0.033/sec0.0308/sec
> total: 129
> txn_try_again  0.0/sec 0.000/sec0.0003/sec
> total: 1
> poll_create_node   0.4/sec 1.933/sec1.9575/sec
> total: 55611
> poll_zero_timeout  0.0/sec 0.067/sec0.0556/sec
> total: 241
> rconn_queued   0.0/sec 0.050/sec0.0594/sec
> total: 1208720
> rconn_sent 0.0/sec 0.050/sec0.0594/sec
> total: 1208720
> seq_change 0.2/sec 0.783/sec0.7492/sec
> total: 13962
> pstream_open   0.0/sec 0.000/sec0./sec
> total: 1
> stream_open0.0/sec 0.000/sec0.0003/sec
> total: 5
> unixctl_received   0.0/sec 0.000/sec0.0011/sec
> total: 4
> unixctl_replied0.0/sec 0.000/sec0.0011/sec
> total: 4
> util_xalloc0.8/sec 1396586.967/sec   240916.6047/sec
> total: 5834154064
> vconn_open 0.0/sec 0.000/sec0./sec
> total: 2
> vconn_received 0.0/sec 0.050/sec0.0594/sec
> total: 632
> vconn_sent 0.0/sec 0.050/sec0.0494/sec
> total: 1213248
> netlink_received   0.0/sec 0.300/sec0.2900/sec
> total: 1188
> netlink_recv_jumbo 0.0/sec 0.083/sec0.0725/sec
> total: 296
> netlink_sent   0.0/sec 0.300/sec0.2900/sec
> total: 1188
> cmap_expand0.0/sec 0.000/sec0./sec
> total: 3
> 82 events never hit
> [root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t
> /run/ovn/ovn-controller.6.ctl coverage/show Event coverage, avg rate
> over last: 5 seconds, last minute, last hour,  hash=d0107601:
> lflow_run  0.2/sec 0.083/sec0.0717/sec
> total: 296
> miniflow_malloc  122834.2/sec 51180.917/sec43930.2869/sec
> total: 181249574
> hindex_pathological0.0/sec 0.000/sec0./sec
> total: 7187
> hindex_expand  0.0/sec 0.000/sec0./sec
> total: 17
> hmap_pathological 13.2/sec 4.967/sec4.1264/sec
> total: 25157
> hmap_expand  14982.2/sec  6205.067/sec 5317.9547/sec
> total: 23755649
> txn_unchanged  1.4/sec 0.400/sec0.3144/sec
> total: 11031
> txn_incomplete 0.4/sec 0.117/sec0.0825/sec
> total: 976
> txn_success0.2/sec 0.050/sec0.0306/sec
> total: 130
> txn_try_again  0.0/sec 0.000/sec0.0003/sec
> total: 1
> poll_create_node   7.6/sec 2.467/sec1.9353/sec
> total: 55649
> poll_zero_timeout  0.4/sec 0.100/sec0.0547/sec
> total: 243
> rconn_queued   0.4/sec 0.083/sec0.0592/sec
> total: 1208722
> rconn_sent 0.4/sec 0.083/sec0.0592/sec
> total: 1208722
> seq_change 2.2/sec 0.900/sec0.7394/sec
> total: 13973
> pstream_open   0.0/sec 0.000/sec0./sec
> total: 1
> stream_open0.0/sec 0.000/sec0.0003/sec
> total: 5
> unixctl_received   0.2/sec 0.017/sec0.0014/sec
> total: 5
> unixctl_replied0.2/sec 0.017/sec0.0014/sec

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Here is the outpu.

[root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t 
/run/ovn/ovn-controller.6.ctl coverage/show
Event coverage, avg rate over last: 5 seconds, last minute, last hour,  
hash=e70a83c8:
lflow_run  0.0/sec 0.083/sec0.0725/sec   total: 295
miniflow_malloc0.0/sec 44356.817/sec44527.3975/sec   total: 
180635403
hindex_pathological0.0/sec 0.000/sec0./sec   total: 7187
hindex_expand  0.0/sec 0.000/sec0./sec   total: 17
hmap_pathological  0.0/sec 4.167/sec4.1806/sec   total: 
25091
hmap_expand0.0/sec  5366.500/sec 5390.0800/sec   total: 
23680738
txn_unchanged  0.0/sec 0.300/sec0.3175/sec   total: 
11024
txn_incomplete 0.0/sec 0.100/sec0.0836/sec   total: 974
txn_success0.0/sec 0.033/sec0.0308/sec   total: 129
txn_try_again  0.0/sec 0.000/sec0.0003/sec   total: 1
poll_create_node   0.4/sec 1.933/sec1.9575/sec   total: 
55611
poll_zero_timeout  0.0/sec 0.067/sec0.0556/sec   total: 241
rconn_queued   0.0/sec 0.050/sec0.0594/sec   total: 
1208720
rconn_sent 0.0/sec 0.050/sec0.0594/sec   total: 
1208720
seq_change 0.2/sec 0.783/sec0.7492/sec   total: 
13962
pstream_open   0.0/sec 0.000/sec0./sec   total: 1
stream_open0.0/sec 0.000/sec0.0003/sec   total: 5
unixctl_received   0.0/sec 0.000/sec0.0011/sec   total: 4
unixctl_replied0.0/sec 0.000/sec0.0011/sec   total: 4
util_xalloc0.8/sec 1396586.967/sec   240916.6047/sec   total: 
5834154064
vconn_open 0.0/sec 0.000/sec0./sec   total: 2
vconn_received 0.0/sec 0.050/sec0.0594/sec   total: 632
vconn_sent 0.0/sec 0.050/sec0.0494/sec   total: 
1213248
netlink_received   0.0/sec 0.300/sec0.2900/sec   total: 1188
netlink_recv_jumbo 0.0/sec 0.083/sec0.0725/sec   total: 296
netlink_sent   0.0/sec 0.300/sec0.2900/sec   total: 1188
cmap_expand0.0/sec 0.000/sec0./sec   total: 3
82 events never hit
[root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t 
/run/ovn/ovn-controller.6.ctl coverage/show
Event coverage, avg rate over last: 5 seconds, last minute, last hour,  
hash=d0107601:
lflow_run  0.2/sec 0.083/sec0.0717/sec   total: 296
miniflow_malloc  122834.2/sec 51180.917/sec43930.2869/sec   total: 
181249574
hindex_pathological0.0/sec 0.000/sec0./sec   total: 7187
hindex_expand  0.0/sec 0.000/sec0./sec   total: 17
hmap_pathological 13.2/sec 4.967/sec4.1264/sec   total: 
25157
hmap_expand  14982.2/sec  6205.067/sec 5317.9547/sec   total: 
23755649
txn_unchanged  1.4/sec 0.400/sec0.3144/sec   total: 
11031
txn_incomplete 0.4/sec 0.117/sec0.0825/sec   total: 976
txn_success0.2/sec 0.050/sec0.0306/sec   total: 130
txn_try_again  0.0/sec 0.000/sec0.0003/sec   total: 1
poll_create_node   7.6/sec 2.467/sec1.9353/sec   total: 
55649
poll_zero_timeout  0.4/sec 0.100/sec0.0547/sec   total: 243
rconn_queued   0.4/sec 0.083/sec0.0592/sec   total: 
1208722
rconn_sent 0.4/sec 0.083/sec0.0592/sec   total: 
1208722
seq_change 2.2/sec 0.900/sec0.7394/sec   total: 
13973
pstream_open   0.0/sec 0.000/sec0./sec   total: 1
stream_open0.0/sec 0.000/sec0.0003/sec   total: 5
unixctl_received   0.2/sec 0.017/sec0.0014/sec   total: 5
unixctl_replied0.2/sec 0.017/sec0.0014/sec   total: 5
util_xalloc  3870474.6/sec 1611767.833/sec   222119.2950/sec   
total: 5853506437
vconn_open 0.0/sec 0.000/sec0./sec   total: 2
vconn_received 0.4/sec 0.083/sec0.0592/sec   total: 634
vconn_sent 0.4/sec 0.083/sec0.0492/sec   total: 
1213250
netlink_received   0.8/sec 0.333/sec0.2861/sec   total: 1192
netlink_recv_jumbo 0.2/sec 0.083/sec0.0717/sec   total: 297
netlink_sent   0.8/sec 0.333/sec0.2861/sec   total: 1192
cmap_expand0.0/sec 0.000/sec0./sec   total: 3
82 events never hit
[root@gateway-1 ~]# docker exec ovn_controller ovs-appctl -t 
/run/ovn/ovn-controller.6.ctl coverage/show
Event coverage, avg 

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Han Zhou]

On Fri, Aug 7, 2020 at 12:57 PM Tony Liu  wrote:

> Enabled debug logging, there are tons of messages.
> Note there are 4353 datapath bindings and 13078 port bindings in SB.
> 4097 LS, 8470 LSP, 256 LR and 4352 LRP in NB. Every 16 LS connect to
> a router. All routers connect to the external network.
>
> ovn-controller on compute node is good. The ovn-controller on gateway
> node is taking 100% cpu. It's probably related to the ports on the
> external network? Any specific messages I need to check?
>
> Any hint to look into it is appreciated!
>
>
If it happens only on gateway, it may be related to ARP handling. Could you
share the output of ovn-appctl -t ovn-controller coverage/show, with 2 - 3
runs in 5s interval?
For the debug log, I'd first check if there is any OVSDB notification from
SB DB, and if yes, what are the changes.

>
> Thanks!
>
> Tony
> > -Original Message-
> > From: Han Zhou 
> > Sent: Friday, August 7, 2020 12:39 PM
> > To: Tony Liu 
> > Cc: ovs-discuss ; ovs-dev  > d...@openvswitch.org>
> > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> > changes in sb-db
> >
> >
> >
> > On Fri, Aug 7, 2020 at 12:35 PM Tony Liu  >  > wrote:
> >
> >
> >   Inline...
> >
> >   Thanks!
> >
> >   Tony
> >   > -Original Message-
> >   > From: Han Zhou mailto:zhou...@gmail.com> >
> >   > Sent: Friday, August 7, 2020 12:29 PM
> >   > To: Tony Liu  >  >
> >   > Cc: ovs-discuss mailto:ovs-
> > disc...@openvswitch.org> >; ovs-dev  >   > d...@openvswitch.org  >
> >   > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu
> > while no
> >   > changes in sb-db
> >   >
> >   >
> >   >
> >   > On Fri, Aug 7, 2020 at 12:19 PM Tony Liu <
> tonyliu0...@hotmail.com
> > 
> >   >  >  > > wrote:
> >   >
> >   >
> >   >   ovn-controller is using UNIX socket connecting to local
> > ovsdb-
> >   > server.
> >   >
> >   > From the log you were showing, you were using tcp:127.0.0.1:6640
> > 
> >
> >   Sorry, what I meant was, given your advice, I just made the change
> > for
> >   ovn-controller to use UNIX socket.
> >
> >
> >
> > Oh, I see, no worries.
> >
> >
> >   >   to connect the local ovsdb.
> >   > >   2020-08-
> > 07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> > 
> >   > >   : connection
> > dropped
> >   > > (Broken pipe)
> >   >
> >   >
> >   >   Inactivity probe doesn't seem to be the cause of high cpu
> > usage.
> >   >
> >   >   The wakeup on connection to sb-db is always followed by a
> >   > "unreasonably
> >   >   long" warning. I guess the pollin event loop is stuck for
> > too long,
> >   > like
> >   >   10s as below.
> >   >   
> >   >   2020-08-07T18:46:49.301Z|00296|poll_loop|INFO|wakeup due to
> > [POLLIN]
> >   > on fd 19 (10.6.20.91:60712 
> >  <->10.6.20.86:6642 
> >   >  ) at lib/stream-fd.c:157 (99% CPU
> usage)
> >   >   2020-08-07T18:46:59.460Z|00297|timeval|WARN|Unreasonably
> > long
> >   > 10153ms poll interval (10075ms user, 1ms system)
> >   >   
> >   >
> >   >   Could that stuck loop be the cause of high cpu usage?
> >   >   What is it polling in?
> >   >   Why is it stuck, waiting for message from sb-db?
> >   >   Isn't it supposed to release the cpu while waiting?
> >   >
> >   >
> >   >
> >   > This log means there are messages received from 10.6.20.86:6642
> > 
> >   >   (the SB DB). Is there SB change? The
> > CPU is
> >   > spent on handling the SB change. Some type of SB changes are not
> > handled
> >   > incrementally.
> >
> >   SB update is driven by ovn-northd in case anything changed in NB,
> >   and ovn-controller in case anything changed on chassis. No, there
> >   is nothing changed in NB, neither chassis.
> >
> >   Should I bump logging level up to dbg? Is that going to show me
> >   what messages ovn-controller is handling?
> >
> >
> >
> > Yes, debug log should show the details.
> >
> >
> >
> >   >
> >   >   Thanks!
> >   >
> >   >   Tony
> >   >
> >   >   > -Original Message-
> >   >   > From: Han Zhou  >    >  > >
> >   >   > Sent: Friday, August 7, 2020 10:32 AM
> >   >   > To: Tony Liu  > 

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Han Zhou]

On Fri, Aug 7, 2020 at 12:35 PM Numan Siddique  wrote:

>
>
> On Sat, Aug 8, 2020 at 12:16 AM Han Zhou  wrote:
>
>>
>>
>> On Thu, Aug 6, 2020 at 10:22 AM Han Zhou  wrote:
>>
>>>
>>>
>>> On Thu, Aug 6, 2020 at 9:15 AM Numan Siddique  wrote:
>>>


 On Thu, Aug 6, 2020 at 9:25 PM Venugopal Iyer 
 wrote:

> Hi, Han:
>
>
>
> A comment inline:
>
>
>
> *From:* ovn-kuberne...@googlegroups.com <
> ovn-kuberne...@googlegroups.com> *On Behalf Of *Han Zhou
> *Sent:* Wednesday, August 5, 2020 3:36 PM
> *To:* Winson Wang 
> *Cc:* ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com;
> Dumitru Ceara ; Han Zhou 
> *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process
> keep the previous Open Flow in br-int
>
>
>
> *External email: Use caution opening links or attachments*
>
>
>
>
>
>
>
> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
> wrote:
>
> Hello OVN Experts,
>
>
> With ovn-k8s,  we need to keep the flows always on br-int which needed
> by running pods on the k8s node.
>
> Is there an ongoing project to address this problem?
>
> If not,  I have one proposal not sure if it is doable.
>
> Please share your thoughts.
> The issue:
>
> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
> every K8s node.  When we restart ovn-controller for upgrade using
> `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic 
> still
> works fine since br-int with flows still be Installed.
>
>
>
> However, when a new ovn-controller starts it will connect OVS IDL and
> do an engine init run,  clearing all OpenFlow flows and install flows 
> based
> on SB DB.
>
> With open flows count above 200K+,  it took more than 15 seconds to
> get all the flows installed br-int bridge again.
>
>
> Proposal solution for the issue:
>
> When the ovn-controller gets “exit --start”,  it will write a
> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
> external-ids column. When new ovn-controller starts, it will check if the
> “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
> OVS IDL to decide if it will force a recomputing process?
>
>
>
>
>
> Hi Winson,
>
>
>
> Thanks for the proposal. Yes, the connection break during upgrading is
> a real issue in a large scale environment. However, the proposal doesn't
> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
> which is a completely different connection from the ovs-vswitchd open-flow
> connection.
>
> To avoid clearing the open-flow table during ovn-controller startup,
> we can find a way to postpone clearing the OVS flows after the recomputing
> in ovn-controller is completed, right before ovn-controller replacing with
> the new flows.
>
> *[vi> ] *
>
> *[vi> ] Seems like we force recompute today if the OVS IDL is
> reconnected. Would it be possible to defer *
>
> *decision to  recompute the flows based on  the  SB’s nb_cfg we have
>  sync’d with? i.e.  If  our nb_cfg is *
>
> *in sync with the SB’s global nb_cfg, we can skip the recompute?  At
> least if nothing has changed since*
>
> *the restart, we won’t need to do anything.. We could stash nb_cfg in
> OVS (once ovn-controller receives*
>
> *conformation from OVS that the physical flows for an nb_cfg update
> are in place), which should be cleared if *
>
> *OVS itself is restarted.. (I mean currently, nb_cfg is used to check
> if NB, SB and Chassis are in sync, we *
>
> *could extend this to OVS/physical flows?)*
>
>
>
> *Have not thought through this though .. so maybe I am missing
> something…*
>
>
>
> *Thanks,*
>
>
>
> *-venu*
>
> This should largely reduce the time of connection broken during
> upgrading. Some changes in the ofctrl module's state machine are required,
> but I am not 100% sure if this approach is applicable. Need to check more
> details.
>


 We can also think if its possible to do the below way
- When ovn-controller starts, it will not clear the flows, but
 instead will get the dump of flows  from the br-int and populate these
 flows in its installed flows
 - And then when it connects to the SB DB and computes the desired
 flows, it will anyway sync up with the installed flows with the desired
 flows
 - And if there is no difference between desired flows and installed
 flows, there will be no impact on the datapath at all.

 Although this would require a careful thought and proper handling.

>>>
>>> Numan, as I responded 

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Enabled debug logging, there are tons of messages.
Note there are 4353 datapath bindings and 13078 port bindings in SB.
4097 LS, 8470 LSP, 256 LR and 4352 LRP in NB. Every 16 LS connect to
a router. All routers connect to the external network.

ovn-controller on compute node is good. The ovn-controller on gateway
node is taking 100% cpu. It's probably related to the ports on the
external network? Any specific messages I need to check?

Any hint to look into it is appreciated!


Thanks!

Tony
> -Original Message-
> From: Han Zhou 
> Sent: Friday, August 7, 2020 12:39 PM
> To: Tony Liu 
> Cc: ovs-discuss ; ovs-dev  d...@openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> changes in sb-db
> 
> 
> 
> On Fri, Aug 7, 2020 at 12:35 PM Tony Liu   > wrote:
> 
> 
>   Inline...
> 
>   Thanks!
> 
>   Tony
>   > -Original Message-
>   > From: Han Zhou mailto:zhou...@gmail.com> >
>   > Sent: Friday, August 7, 2020 12:29 PM
>   > To: Tony Liu   >
>   > Cc: ovs-discuss mailto:ovs-
> disc...@openvswitch.org> >; ovs-dev> d...@openvswitch.org  >
>   > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu
> while no
>   > changes in sb-db
>   >
>   >
>   >
>   > On Fri, Aug 7, 2020 at 12:19 PM Tony Liu  
>   >   > > wrote:
>   >
>   >
>   >   ovn-controller is using UNIX socket connecting to local
> ovsdb-
>   > server.
>   >
>   > From the log you were showing, you were using tcp:127.0.0.1:6640
> 
> 
>   Sorry, what I meant was, given your advice, I just made the change
> for
>   ovn-controller to use UNIX socket.
> 
> 
> 
> Oh, I see, no worries.
> 
> 
>   >   to connect the local ovsdb.
>   > >   2020-08-
> 07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> 
>   > >   : connection
> dropped
>   > > (Broken pipe)
>   >
>   >
>   >   Inactivity probe doesn't seem to be the cause of high cpu
> usage.
>   >
>   >   The wakeup on connection to sb-db is always followed by a
>   > "unreasonably
>   >   long" warning. I guess the pollin event loop is stuck for
> too long,
>   > like
>   >   10s as below.
>   >   
>   >   2020-08-07T18:46:49.301Z|00296|poll_loop|INFO|wakeup due to
> [POLLIN]
>   > on fd 19 (10.6.20.91:60712 
>  <->10.6.20.86:6642 
>   >  ) at lib/stream-fd.c:157 (99% CPU usage)
>   >   2020-08-07T18:46:59.460Z|00297|timeval|WARN|Unreasonably
> long
>   > 10153ms poll interval (10075ms user, 1ms system)
>   >   
>   >
>   >   Could that stuck loop be the cause of high cpu usage?
>   >   What is it polling in?
>   >   Why is it stuck, waiting for message from sb-db?
>   >   Isn't it supposed to release the cpu while waiting?
>   >
>   >
>   >
>   > This log means there are messages received from 10.6.20.86:6642
> 
>   >   (the SB DB). Is there SB change? The
> CPU is
>   > spent on handling the SB change. Some type of SB changes are not
> handled
>   > incrementally.
> 
>   SB update is driven by ovn-northd in case anything changed in NB,
>   and ovn-controller in case anything changed on chassis. No, there
>   is nothing changed in NB, neither chassis.
> 
>   Should I bump logging level up to dbg? Is that going to show me
>   what messages ovn-controller is handling?
> 
> 
> 
> Yes, debug log should show the details.
> 
> 
> 
>   >
>   >   Thanks!
>   >
>   >   Tony
>   >
>   >   > -Original Message-
>   >   > From: Han Zhou      > >
>   >   > Sent: Friday, August 7, 2020 10:32 AM
>   >   > To: Tony Liu  
>   >   > >
>   >   > Cc: ovs-discuss mailto:ovs-
> disc...@openvswitch.org>  
>   > disc...@openvswitch.org  > >;
> ovs-dev>   > d...@openvswitch.org 
>  > >
>   >   > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100%
> cpu
>   > while no
>   >   > changes in sb-db
>   >   >
>   >   >
>   >   >
>   >   > On Fri, Aug 

Re: [ovs-discuss] [OVN] no response to inactivity probe

[Tony Liu]

Good to know, thanks!

Tony
> -Original Message-
> From: Han Zhou 
> Sent: Friday, August 7, 2020 12:36 PM
> To: Tony Liu 
> Cc: Han Zhou ; Numan Siddique ; ovs-dev
> ; ovs-discuss 
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 
> The raft probe is disabled if you use the latest version of OVS, e.g.
> 2.13.1.
> 
> 
> On Fri, Aug 7, 2020 at 12:28 PM Tony Liu   > wrote:
> 
> 
>   Another one here, there is inactivity probe on the raft cluster
>   port.
>   
>   2020-08-07T19:04:53.184Z|02735|reconnect|ERR|tcp:10.6.20.85:6644
>  : no response to inactivity probe after 5
> seconds, disconnecting
>   2020-08-07T19:04:53.184Z|02736|reconnect|INFO|tcp:10.6.20.85:6644
>  : connection dropped
>   2020-08-07T19:04:54.185Z|02737|reconnect|INFO|tcp:10.6.20.85:6644
>  : connecting...
>   2020-08-07T19:04:54.185Z|02738|reconnect|INFO|tcp:10.6.20.85:6644
>  : connected
>   2020-08-07T19:15:26.228Z|02739|reconnect|ERR|tcp:10.6.20.84:49440
>  : no response to inactivity probe after 5
> seconds, disconnecting
>   2020-08-07T19:15:26.769Z|02740|reconnect|ERR|tcp:10.6.20.84:6644
>  : no response to inactivity probe after 5
> seconds, disconnecting
>   2020-08-07T19:15:26.769Z|02741|reconnect|INFO|tcp:10.6.20.84:6644
>  : connection dropped
>   2020-08-07T19:15:27.771Z|02742|reconnect|INFO|tcp:10.6.20.84:6644
>  : connecting...
>   2020-08-07T19:15:27.771Z|02743|reconnect|INFO|tcp:10.6.20.84:6644
>  : connected
>   
>   Which configuration is for that probe interval?
> 
> 
>   Thanks!
> 
>   Tony
> 
>   > -Original Message-
>   > From: dev mailto:ovs-dev-
> boun...@openvswitch.org> > On Behalf Of Tony Liu
>   > Sent: Thursday, August 6, 2020 7:45 PM
>   > To: Han Zhou mailto:hz...@ovn.org> >; Numan
> Siddique mailto:num...@ovn.org> >
>   > Cc: ovs-dev mailto:ovs-
> d...@openvswitch.org> >; ovs-discuss> disc...@openvswitch.org  >
>   > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to
> inactivity
>   > probe
>   >
>   > Hi Han and Numan,
>   >
>   > I'd like to have a few more clarifications.
>   >
>   > For inactivity probe:
>   > From ovn-controller to ovn-sb-db: ovn-remote-probe-interval
>   >
>   > From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval
>   >
>   > From ovn-controller to local ovsdb: which interval?
>   >
>   > From local ovsdb to ovn-controller: which interval?
>   >
>   > From ovs-vswitchd to ovn-controller: which interval?
>   >
>   >
>   > Regarding to the connection between ovn-controller and local
> ovsdb-
>   > server, I recall that UNIX socket is lighter than TCP socket and
> UNIX
>   > socket is recommended for local communication.
>   > Is that right?
>   >
>   >
>   > Thanks!
>   >
>   > Tony
>   >
>   > > -Original Message-
>   > > From: Han Zhou mailto:hz...@ovn.org> >
>   > > Sent: Thursday, August 6, 2020 12:42 PM
>   > > To: Tony Liu   >
>   > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan
> Siddique mailto:num...@ovn.org> >; ovs-dev
>   > > mailto:ovs-...@openvswitch.org> >;
> ovs-discuss mailto:ovs-
> disc...@openvswitch.org> >
>   > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> probe
>   > >
>   > >
>   > >
>   > > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu
> mailto:tonyliu0...@hotmail.com>
>   > >   > > wrote:
>   > > >
>   > > > Inline...
>   > > >
>   > > > Thanks!
>   > > >
>   > > > Tony
>   > > > > -Original Message-
>   > > > > From: Han Zhou mailto:hz...@ovn.org>
>  > >
>   > > > > Sent: Thursday, August 6, 2020 11:37 AM
>   > > > > To: Tony Liu  
>   > > > >   > >
>   > > > > Cc: Han Zhou mailto:hz...@ovn.org>
>  > >; Numan
>   > > > > Siddique mailto:num...@ovn.org>
>  > >; ovs-dev
>   > > > > mailto:ovs-...@openvswitch.org>
>  > >;
>   > > > > ovs-discuss mailto:ovs-
> disc...@openvswitch.org>
>   > > > >  disc...@openvswitch.org> > >
>   > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> probe
>   > > > >
>   > > > >
> 

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Inline...

Thanks!

Tony
> -Original Message-
> From: Han Zhou 
> Sent: Friday, August 7, 2020 12:29 PM
> To: Tony Liu 
> Cc: ovs-discuss ; ovs-dev  d...@openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> changes in sb-db
> 
> 
> 
> On Fri, Aug 7, 2020 at 12:19 PM Tony Liu   > wrote:
> 
> 
>   ovn-controller is using UNIX socket connecting to local ovsdb-
> server.
> 
> From the log you were showing, you were using tcp:127.0.0.1:6640

Sorry, what I meant was, given your advice, I just made the change for
ovn-controller to use UNIX socket.

>   to connect the local ovsdb.
> >   2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> >   : connection dropped
> > (Broken pipe)
> 
> 
>   Inactivity probe doesn't seem to be the cause of high cpu usage.
> 
>   The wakeup on connection to sb-db is always followed by a
> "unreasonably
>   long" warning. I guess the pollin event loop is stuck for too long,
> like
>   10s as below.
>   
>   2020-08-07T18:46:49.301Z|00296|poll_loop|INFO|wakeup due to [POLLIN]
> on fd 19 (10.6.20.91:60712  <->10.6.20.86:6642
>  ) at lib/stream-fd.c:157 (99% CPU usage)
>   2020-08-07T18:46:59.460Z|00297|timeval|WARN|Unreasonably long
> 10153ms poll interval (10075ms user, 1ms system)
>   
> 
>   Could that stuck loop be the cause of high cpu usage?
>   What is it polling in?
>   Why is it stuck, waiting for message from sb-db?
>   Isn't it supposed to release the cpu while waiting?
> 
> 
> 
> This log means there are messages received from 10.6.20.86:6642
>   (the SB DB). Is there SB change? The CPU is
> spent on handling the SB change. Some type of SB changes are not handled
> incrementally.

SB update is driven by ovn-northd in case anything changed in NB,
and ovn-controller in case anything changed on chassis. No, there
is nothing changed in NB, neither chassis.

Should I bump logging level up to dbg? Is that going to show me
what messages ovn-controller is handling?

> 
>   Thanks!
> 
>   Tony
> 
>   > -Original Message-
>   > From: Han Zhou mailto:zhou...@gmail.com> >
>   > Sent: Friday, August 7, 2020 10:32 AM
>   > To: Tony Liu   >
>   > Cc: ovs-discuss mailto:ovs-
> disc...@openvswitch.org> >; ovs-dev> d...@openvswitch.org  >
>   > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu
> while no
>   > changes in sb-db
>   >
>   >
>   >
>   > On Fri, Aug 7, 2020 at 10:05 AM Tony Liu  
>   >   > > wrote:
>   >
>   >
>   >   Hi,
>   >
>   >   Here are some logging snippets from ovn-controller.
>   >   
>   >   2020-08-07T16:38:04.020Z|29250|timeval|WARN|Unreasonably
> long
>   > 8954ms poll interval (8895ms user, 0ms system)
>   >   
>   >   What's that mean? Is it harmless?
>   >
>   >   
>   >   2020-08-07T16:38:04.021Z|29251|timeval|WARN|context
> switches: 0
>   > voluntary, 6 involuntary
>   >   2020-08-07T16:38:04.022Z|29252|poll_loop|INFO|wakeup due to
> [POLLIN]
>   > on fd 19 (10.6.20.91:60398 
>  <->10.6.20.86:6642 
>   >  ) at lib/stream-fd.c:157 (99% CPU usage)
>   >   
>   >   Is this wakeup caused by changes in sb-db?
>   >   Why is ovn-controller so busy?
>   >
>   >   
>   >   2020-08-
> 07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> 
>   >  : connection dropped (Broken pipe)
>   >   
>   >   Connection to local ovsdb-server is dropped.
>   >   Is this caused by the timeout of inactivity probe?
>   >
>   >   
>   >   2020-08-07T16:38:04.035Z|29254|poll_loop|INFO|wakeup due to
> [POLLIN]
>   > on fd 20 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-
> fd.c:157
>   > (99% CPU usage)
>   >   
>   >   What causes this wakeup?
>   >
>   >   
>   >   2020-08-07T16:38:04.048Z|29255|poll_loop|INFO|wakeup due to
> 0-ms
>   > timeout at lib/ovsdb-idl.c:5391 (99% CPU usage)
>   >   
>   >   What's this 0-ms wakeup mean?
>   >
>   >   
>   >   2020-08-07T16:38:05.022Z|29256|poll_loop|INFO|wakeup due to
> 962-ms
>   > timeout at lib/reconnect.c:643 (99% CPU usage)
>   >   2020-08-
> 

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Han Zhou]

On Fri, Aug 7, 2020 at 12:35 PM Tony Liu  wrote:

> Inline...
>
> Thanks!
>
> Tony
> > -Original Message-
> > From: Han Zhou 
> > Sent: Friday, August 7, 2020 12:29 PM
> > To: Tony Liu 
> > Cc: ovs-discuss ; ovs-dev  > d...@openvswitch.org>
> > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> > changes in sb-db
> >
> >
> >
> > On Fri, Aug 7, 2020 at 12:19 PM Tony Liu  >  > wrote:
> >
> >
> >   ovn-controller is using UNIX socket connecting to local ovsdb-
> > server.
> >
> > From the log you were showing, you were using tcp:127.0.0.1:6640
>
> Sorry, what I meant was, given your advice, I just made the change for
> ovn-controller to use UNIX socket.
>
> Oh, I see, no worries.

>   to connect the local ovsdb.
> > >   2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> > >   : connection dropped
> > > (Broken pipe)
> >
> >
> >   Inactivity probe doesn't seem to be the cause of high cpu usage.
> >
> >   The wakeup on connection to sb-db is always followed by a
> > "unreasonably
> >   long" warning. I guess the pollin event loop is stuck for too long,
> > like
> >   10s as below.
> >   
> >   2020-08-07T18:46:49.301Z|00296|poll_loop|INFO|wakeup due to
> [POLLIN]
> > on fd 19 (10.6.20.91:60712  <->10.6.20.86:6642
> >  ) at lib/stream-fd.c:157 (99% CPU usage)
> >   2020-08-07T18:46:59.460Z|00297|timeval|WARN|Unreasonably long
> > 10153ms poll interval (10075ms user, 1ms system)
> >   
> >
> >   Could that stuck loop be the cause of high cpu usage?
> >   What is it polling in?
> >   Why is it stuck, waiting for message from sb-db?
> >   Isn't it supposed to release the cpu while waiting?
> >
> >
> >
> > This log means there are messages received from 10.6.20.86:6642
> >   (the SB DB). Is there SB change? The CPU is
> > spent on handling the SB change. Some type of SB changes are not handled
> > incrementally.
>
> SB update is driven by ovn-northd in case anything changed in NB,
> and ovn-controller in case anything changed on chassis. No, there
> is nothing changed in NB, neither chassis.
>
> Should I bump logging level up to dbg? Is that going to show me
> what messages ovn-controller is handling?
>
> Yes, debug log should show the details.


> >
> >   Thanks!
> >
> >   Tony
> >
> >   > -Original Message-
> >   > From: Han Zhou mailto:zhou...@gmail.com> >
> >   > Sent: Friday, August 7, 2020 10:32 AM
> >   > To: Tony Liu  >  >
> >   > Cc: ovs-discuss mailto:ovs-
> > disc...@openvswitch.org> >; ovs-dev  >   > d...@openvswitch.org  >
> >   > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu
> > while no
> >   > changes in sb-db
> >   >
> >   >
> >   >
> >   > On Fri, Aug 7, 2020 at 10:05 AM Tony Liu <
> tonyliu0...@hotmail.com
> > 
> >   >  >  > > wrote:
> >   >
> >   >
> >   >   Hi,
> >   >
> >   >   Here are some logging snippets from ovn-controller.
> >   >   
> >   >   2020-08-07T16:38:04.020Z|29250|timeval|WARN|Unreasonably
> > long
> >   > 8954ms poll interval (8895ms user, 0ms system)
> >   >   
> >   >   What's that mean? Is it harmless?
> >   >
> >   >   
> >   >   2020-08-07T16:38:04.021Z|29251|timeval|WARN|context
> > switches: 0
> >   > voluntary, 6 involuntary
> >   >   2020-08-07T16:38:04.022Z|29252|poll_loop|INFO|wakeup due to
> > [POLLIN]
> >   > on fd 19 (10.6.20.91:60398 
> >  <->10.6.20.86:6642 
> >   >  ) at lib/stream-fd.c:157 (99% CPU
> usage)
> >   >   
> >   >   Is this wakeup caused by changes in sb-db?
> >   >   Why is ovn-controller so busy?
> >   >
> >   >   
> >   >   2020-08-
> > 07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> > 
> >   >  : connection dropped (Broken pipe)
> >   >   
> >   >   Connection to local ovsdb-server is dropped.
> >   >   Is this caused by the timeout of inactivity probe?
> >   >
> >   >   
> >   >   2020-08-07T16:38:04.035Z|29254|poll_loop|INFO|wakeup due to
> > [POLLIN]
> >   > on fd 20 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-
> > fd.c:157
> >   > (99% CPU usage)
> >   >   
> >   >   What causes this wakeup?
> >   >
> >   >   
> >   >   

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Numan Siddique]

On Sat, Aug 8, 2020 at 12:16 AM Han Zhou  wrote:

>
>
> On Thu, Aug 6, 2020 at 10:22 AM Han Zhou  wrote:
>
>>
>>
>> On Thu, Aug 6, 2020 at 9:15 AM Numan Siddique  wrote:
>>
>>>
>>>
>>> On Thu, Aug 6, 2020 at 9:25 PM Venugopal Iyer 
>>> wrote:
>>>
 Hi, Han:



 A comment inline:



 *From:* ovn-kuberne...@googlegroups.com <
 ovn-kuberne...@googlegroups.com> *On Behalf Of *Han Zhou
 *Sent:* Wednesday, August 5, 2020 3:36 PM
 *To:* Winson Wang 
 *Cc:* ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com;
 Dumitru Ceara ; Han Zhou 
 *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process
 keep the previous Open Flow in br-int



 *External email: Use caution opening links or attachments*







 On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
 wrote:

 Hello OVN Experts,


 With ovn-k8s,  we need to keep the flows always on br-int which needed
 by running pods on the k8s node.

 Is there an ongoing project to address this problem?

 If not,  I have one proposal not sure if it is doable.

 Please share your thoughts.
 The issue:

 In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
 every K8s node.  When we restart ovn-controller for upgrade using
 `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
 works fine since br-int with flows still be Installed.



 However, when a new ovn-controller starts it will connect OVS IDL and
 do an engine init run,  clearing all OpenFlow flows and install flows based
 on SB DB.

 With open flows count above 200K+,  it took more than 15 seconds to get
 all the flows installed br-int bridge again.


 Proposal solution for the issue:

 When the ovn-controller gets “exit --start”,  it will write a
 “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
 external-ids column. When new ovn-controller starts, it will check if the
 “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
 OVS IDL to decide if it will force a recomputing process?





 Hi Winson,



 Thanks for the proposal. Yes, the connection break during upgrading is
 a real issue in a large scale environment. However, the proposal doesn't
 work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
 which is a completely different connection from the ovs-vswitchd open-flow
 connection.

 To avoid clearing the open-flow table during ovn-controller startup, we
 can find a way to postpone clearing the OVS flows after the recomputing in
 ovn-controller is completed, right before ovn-controller replacing with the
 new flows.

 *[vi> ] *

 *[vi> ] Seems like we force recompute today if the OVS IDL is
 reconnected. Would it be possible to defer *

 *decision to  recompute the flows based on  the  SB’s nb_cfg we have
  sync’d with? i.e.  If  our nb_cfg is *

 *in sync with the SB’s global nb_cfg, we can skip the recompute?  At
 least if nothing has changed since*

 *the restart, we won’t need to do anything.. We could stash nb_cfg in
 OVS (once ovn-controller receives*

 *conformation from OVS that the physical flows for an nb_cfg update are
 in place), which should be cleared if *

 *OVS itself is restarted.. (I mean currently, nb_cfg is used to check
 if NB, SB and Chassis are in sync, we *

 *could extend this to OVS/physical flows?)*



 *Have not thought through this though .. so maybe I am missing
 something…*



 *Thanks,*



 *-venu*

 This should largely reduce the time of connection broken during
 upgrading. Some changes in the ofctrl module's state machine are required,
 but I am not 100% sure if this approach is applicable. Need to check more
 details.

>>>
>>>
>>> We can also think if its possible to do the below way
>>>- When ovn-controller starts, it will not clear the flows, but
>>> instead will get the dump of flows  from the br-int and populate these
>>> flows in its installed flows
>>> - And then when it connects to the SB DB and computes the desired
>>> flows, it will anyway sync up with the installed flows with the desired
>>> flows
>>> - And if there is no difference between desired flows and installed
>>> flows, there will be no impact on the datapath at all.
>>>
>>> Although this would require a careful thought and proper handling.
>>>
>>
>> Numan, as I responded to Girish, this avoids the time spent on the
>> one-time flow installation after restart (the < 10% part of the connection
>> broken time), but I think currently the major problem is that > 90% of the
>> time is spent on waiting 

Re: [ovs-discuss] [OVN] no response to inactivity probe

[Tony Liu]

Another one here, there is inactivity probe on the raft cluster
port. 

2020-08-07T19:04:53.184Z|02735|reconnect|ERR|tcp:10.6.20.85:6644: no response 
to inactivity probe after 5 seconds, disconnecting
2020-08-07T19:04:53.184Z|02736|reconnect|INFO|tcp:10.6.20.85:6644: connection 
dropped
2020-08-07T19:04:54.185Z|02737|reconnect|INFO|tcp:10.6.20.85:6644: connecting...
2020-08-07T19:04:54.185Z|02738|reconnect|INFO|tcp:10.6.20.85:6644: connected
2020-08-07T19:15:26.228Z|02739|reconnect|ERR|tcp:10.6.20.84:49440: no response 
to inactivity probe after 5 seconds, disconnecting
2020-08-07T19:15:26.769Z|02740|reconnect|ERR|tcp:10.6.20.84:6644: no response 
to inactivity probe after 5 seconds, disconnecting
2020-08-07T19:15:26.769Z|02741|reconnect|INFO|tcp:10.6.20.84:6644: connection 
dropped
2020-08-07T19:15:27.771Z|02742|reconnect|INFO|tcp:10.6.20.84:6644: connecting...
2020-08-07T19:15:27.771Z|02743|reconnect|INFO|tcp:10.6.20.84:6644: connected

Which configuration is for that probe interval?


Thanks!

Tony

> -Original Message-
> From: dev  On Behalf Of Tony Liu
> Sent: Thursday, August 6, 2020 7:45 PM
> To: Han Zhou ; Numan Siddique 
> Cc: ovs-dev ; ovs-discuss  disc...@openvswitch.org>
> Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to inactivity
> probe
> 
> Hi Han and Numan,
> 
> I'd like to have a few more clarifications.
> 
> For inactivity probe:
> From ovn-controller to ovn-sb-db: ovn-remote-probe-interval
> 
> From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval
> 
> From ovn-controller to local ovsdb: which interval?
> 
> From local ovsdb to ovn-controller: which interval?
> 
> From ovs-vswitchd to ovn-controller: which interval?
> 
> 
> Regarding to the connection between ovn-controller and local ovsdb-
> server, I recall that UNIX socket is lighter than TCP socket and UNIX
> socket is recommended for local communication.
> Is that right?
> 
> 
> Thanks!
> 
> Tony
> 
> > -Original Message-
> > From: Han Zhou 
> > Sent: Thursday, August 6, 2020 12:42 PM
> > To: Tony Liu 
> > Cc: Han Zhou ; Numan Siddique ; ovs-dev
> > ; ovs-discuss 
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> >
> >
> > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu  >  > wrote:
> > >
> > > Inline...
> > >
> > > Thanks!
> > >
> > > Tony
> > > > -Original Message-
> > > > From: Han Zhou mailto:hz...@ovn.org> >
> > > > Sent: Thursday, August 6, 2020 11:37 AM
> > > > To: Tony Liu  > > >  >
> > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan
> > > > Siddique mailto:num...@ovn.org> >; ovs-dev
> > > > mailto:ovs-...@openvswitch.org> >;
> > > > ovs-discuss  > > >  >
> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > > >
> > > >
> > > >
> > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu  > > >   >  > > wrote:
> > > > >
> > > > > Inline... (please read with monospaced font:))
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Tony
> > > > > > -Original Message-
> > > > > > From: Han Zhou mailto:hz...@ovn.org>
> > > > > >  > >
> > > > > > Sent: Wednesday, August 5, 2020 11:48 PM
> > > > > > To: Tony Liu  > > > > > 
> > > > > >  > > > > >  > >
> > > > > > Cc: Han Zhou mailto:hz...@ovn.org>
> > > > > >  > >; Numan
> > > > > > Siddique mailto:num...@ovn.org>
> > > > > >  > >; ovs-dev
> > > > > > mailto:ovs-...@openvswitch.org>
> > > > > >  > > > > > 
> > > > > > > >; ovs-discuss  > > > > > 
> > > > > >  > > > > >  > >
> > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> > > > > > probe
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu
> > > > > > mailto:tonyliu0...@hotmail.com>
> > > > > >  > > > > >  >
> > > > > >  > > > > > 
> > > >  >  > > > wrote:
> > > > > >
> > > > > >
> > > > > >   I set the connection target="ptcp:6641:10.6.20.84" for
> > > > > > ovn-nb-
> > > > db
> > > > > >   and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the
> > > > > > first
> > > > node
> > > > > >   of cluster. Also ovn-openflow-probe-interval=30 on
> > > > > > compute
> > > > node.
> > > > > >   It seems helping. Not that many connect/drop/reconnect
> > > > > > in
> > > > logging.
> > > > > >

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Han Zhou]

On Fri, Aug 7, 2020 at 12:19 PM Tony Liu  wrote:

> ovn-controller is using UNIX socket connecting to local ovsdb-server.
>

>From the log you were showing, you were using tcp:127.0.0.1:6640 to connect
the local ovsdb.
>   2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
>  : connection dropped (Broken pipe)


> Inactivity probe doesn't seem to be the cause of high cpu usage.
>
> The wakeup on connection to sb-db is always followed by a "unreasonably
> long" warning. I guess the pollin event loop is stuck for too long, like
> 10s as below.
> 
> 2020-08-07T18:46:49.301Z|00296|poll_loop|INFO|wakeup due to [POLLIN] on fd
> 19 (10.6.20.91:60712<->10.6.20.86:6642) at lib/stream-fd.c:157 (99% CPU
> usage)
> 2020-08-07T18:46:59.460Z|00297|timeval|WARN|Unreasonably long 10153ms poll
> interval (10075ms user, 1ms system)
> 
>
> Could that stuck loop be the cause of high cpu usage?
> What is it polling in?
> Why is it stuck, waiting for message from sb-db?
> Isn't it supposed to release the cpu while waiting?
>
> This log means there are messages received from 10.6.20.86:6642 (the SB
DB). Is there SB change? The CPU is spent on handling the SB change. Some
type of SB changes are not handled incrementally.


> Thanks!
>
> Tony
>
> > -Original Message-
> > From: Han Zhou 
> > Sent: Friday, August 7, 2020 10:32 AM
> > To: Tony Liu 
> > Cc: ovs-discuss ; ovs-dev  > d...@openvswitch.org>
> > Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> > changes in sb-db
> >
> >
> >
> > On Fri, Aug 7, 2020 at 10:05 AM Tony Liu  >  > wrote:
> >
> >
> >   Hi,
> >
> >   Here are some logging snippets from ovn-controller.
> >   
> >   2020-08-07T16:38:04.020Z|29250|timeval|WARN|Unreasonably long
> > 8954ms poll interval (8895ms user, 0ms system)
> >   
> >   What's that mean? Is it harmless?
> >
> >   
> >   2020-08-07T16:38:04.021Z|29251|timeval|WARN|context switches: 0
> > voluntary, 6 involuntary
> >   2020-08-07T16:38:04.022Z|29252|poll_loop|INFO|wakeup due to
> [POLLIN]
> > on fd 19 (10.6.20.91:60398  <->10.6.20.86:6642
> >  ) at lib/stream-fd.c:157 (99% CPU usage)
> >   
> >   Is this wakeup caused by changes in sb-db?
> >   Why is ovn-controller so busy?
> >
> >   
> >   2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
> >  : connection dropped (Broken pipe)
> >   
> >   Connection to local ovsdb-server is dropped.
> >   Is this caused by the timeout of inactivity probe?
> >
> >   
> >   2020-08-07T16:38:04.035Z|29254|poll_loop|INFO|wakeup due to
> [POLLIN]
> > on fd 20 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157
> > (99% CPU usage)
> >   
> >   What causes this wakeup?
> >
> >   
> >   2020-08-07T16:38:04.048Z|29255|poll_loop|INFO|wakeup due to 0-ms
> > timeout at lib/ovsdb-idl.c:5391 (99% CPU usage)
> >   
> >   What's this 0-ms wakeup mean?
> >
> >   
> >   2020-08-07T16:38:05.022Z|29256|poll_loop|INFO|wakeup due to 962-ms
> > timeout at lib/reconnect.c:643 (99% CPU usage)
> >   2020-08-07T16:38:05.023Z|29257|reconnect|INFO|tcp:127.0.0.1:6640
> >  : connecting...
> >   2020-08-07T16:38:05.041Z|29258|poll_loop|INFO|wakeup due to
> > [POLLOUT] on fd 14 (127.0.0.1:51478  <-
> > >127.0.0.1:6640  ) at lib/stream-fd.c:153 (99%
> > CPU usage)
> >   2020-08-07T16:38:05.041Z|29259|reconnect|INFO|tcp:127.0.0.1:6640
> >  : connected
> >   
> >   Retry to connect to local ovsdb-server. A pollout event is
> > triggered
> >   right after connection is established. What's poolout?
> >
> >   ovn-controller is taking 100% CPU now, and there is no changes in
> >   sb-db (not busy). It seems that it's busy with local ovsdb-server
> >   or vswitchd. I'd like to understand why ovn-controller is so busy?
> >   All inactivity probe intervals are set to 30s.
> >
> >
> >
> >
> > Is there change from the local ovsdb? You can enable dbg log to see what
> > is happening.
> > For the local ovsdb probe, I have mentioned in the other thread that
> > UNIX socket is recommended (instead of tcp 127.0.0.1). Using UNIX socket
> > disables probe by default.
> >
> > Thanks,
> > Han
>
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

ovn-controller is using UNIX socket connecting to local ovsdb-server.
Inactivity probe doesn't seem to be the cause of high cpu usage.

The wakeup on connection to sb-db is always followed by a "unreasonably
long" warning. I guess the pollin event loop is stuck for too long, like
10s as below.

2020-08-07T18:46:49.301Z|00296|poll_loop|INFO|wakeup due to [POLLIN] on fd 19 
(10.6.20.91:60712<->10.6.20.86:6642) at lib/stream-fd.c:157 (99% CPU usage)
2020-08-07T18:46:59.460Z|00297|timeval|WARN|Unreasonably long 10153ms poll 
interval (10075ms user, 1ms system)


Could that stuck loop be the cause of high cpu usage?
What is it polling in?
Why is it stuck, waiting for message from sb-db?
Isn't it supposed to release the cpu while waiting?


Thanks!

Tony

> -Original Message-
> From: Han Zhou 
> Sent: Friday, August 7, 2020 10:32 AM
> To: Tony Liu 
> Cc: ovs-discuss ; ovs-dev  d...@openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no
> changes in sb-db
> 
> 
> 
> On Fri, Aug 7, 2020 at 10:05 AM Tony Liu   > wrote:
> 
> 
>   Hi,
> 
>   Here are some logging snippets from ovn-controller.
>   
>   2020-08-07T16:38:04.020Z|29250|timeval|WARN|Unreasonably long
> 8954ms poll interval (8895ms user, 0ms system)
>   
>   What's that mean? Is it harmless?
> 
>   
>   2020-08-07T16:38:04.021Z|29251|timeval|WARN|context switches: 0
> voluntary, 6 involuntary
>   2020-08-07T16:38:04.022Z|29252|poll_loop|INFO|wakeup due to [POLLIN]
> on fd 19 (10.6.20.91:60398  <->10.6.20.86:6642
>  ) at lib/stream-fd.c:157 (99% CPU usage)
>   
>   Is this wakeup caused by changes in sb-db?
>   Why is ovn-controller so busy?
> 
>   
>   2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640
>  : connection dropped (Broken pipe)
>   
>   Connection to local ovsdb-server is dropped.
>   Is this caused by the timeout of inactivity probe?
> 
>   
>   2020-08-07T16:38:04.035Z|29254|poll_loop|INFO|wakeup due to [POLLIN]
> on fd 20 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157
> (99% CPU usage)
>   
>   What causes this wakeup?
> 
>   
>   2020-08-07T16:38:04.048Z|29255|poll_loop|INFO|wakeup due to 0-ms
> timeout at lib/ovsdb-idl.c:5391 (99% CPU usage)
>   
>   What's this 0-ms wakeup mean?
> 
>   
>   2020-08-07T16:38:05.022Z|29256|poll_loop|INFO|wakeup due to 962-ms
> timeout at lib/reconnect.c:643 (99% CPU usage)
>   2020-08-07T16:38:05.023Z|29257|reconnect|INFO|tcp:127.0.0.1:6640
>  : connecting...
>   2020-08-07T16:38:05.041Z|29258|poll_loop|INFO|wakeup due to
> [POLLOUT] on fd 14 (127.0.0.1:51478  <-
> >127.0.0.1:6640  ) at lib/stream-fd.c:153 (99%
> CPU usage)
>   2020-08-07T16:38:05.041Z|29259|reconnect|INFO|tcp:127.0.0.1:6640
>  : connected
>   
>   Retry to connect to local ovsdb-server. A pollout event is
> triggered
>   right after connection is established. What's poolout?
> 
>   ovn-controller is taking 100% CPU now, and there is no changes in
>   sb-db (not busy). It seems that it's busy with local ovsdb-server
>   or vswitchd. I'd like to understand why ovn-controller is so busy?
>   All inactivity probe intervals are set to 30s.
> 
> 
> 
> 
> Is there change from the local ovsdb? You can enable dbg log to see what
> is happening.
> For the local ovsdb probe, I have mentioned in the other thread that
> UNIX socket is recommended (instead of tcp 127.0.0.1). Using UNIX socket
> disables probe by default.
> 
> Thanks,
> Han

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int

[Han Zhou]

On Thu, Aug 6, 2020 at 10:22 AM Han Zhou  wrote:

>
>
> On Thu, Aug 6, 2020 at 9:15 AM Numan Siddique  wrote:
>
>>
>>
>> On Thu, Aug 6, 2020 at 9:25 PM Venugopal Iyer 
>> wrote:
>>
>>> Hi, Han:
>>>
>>>
>>>
>>> A comment inline:
>>>
>>>
>>>
>>> *From:* ovn-kuberne...@googlegroups.com 
>>> *On Behalf Of *Han Zhou
>>> *Sent:* Wednesday, August 5, 2020 3:36 PM
>>> *To:* Winson Wang 
>>> *Cc:* ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com;
>>> Dumitru Ceara ; Han Zhou 
>>> *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process
>>> keep the previous Open Flow in br-int
>>>
>>>
>>>
>>> *External email: Use caution opening links or attachments*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang 
>>> wrote:
>>>
>>> Hello OVN Experts,
>>>
>>>
>>> With ovn-k8s,  we need to keep the flows always on br-int which needed
>>> by running pods on the k8s node.
>>>
>>> Is there an ongoing project to address this problem?
>>>
>>> If not,  I have one proposal not sure if it is doable.
>>>
>>> Please share your thoughts.
>>> The issue:
>>>
>>> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on
>>> every K8s node.  When we restart ovn-controller for upgrade using
>>> `ovs-appctl -t ovn-controller exit --restart`,  the remaining traffic still
>>> works fine since br-int with flows still be Installed.
>>>
>>>
>>>
>>> However, when a new ovn-controller starts it will connect OVS IDL and do
>>> an engine init run,  clearing all OpenFlow flows and install flows based on
>>> SB DB.
>>>
>>> With open flows count above 200K+,  it took more than 15 seconds to get
>>> all the flows installed br-int bridge again.
>>>
>>>
>>> Proposal solution for the issue:
>>>
>>> When the ovn-controller gets “exit --start”,  it will write a
>>> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in
>>> external-ids column. When new ovn-controller starts, it will check if the
>>> “ovs-cond-seqno” exists in the Open_vSwitch table,  and get the seqno from
>>> OVS IDL to decide if it will force a recomputing process?
>>>
>>>
>>>
>>>
>>>
>>> Hi Winson,
>>>
>>>
>>>
>>> Thanks for the proposal. Yes, the connection break during upgrading is a
>>> real issue in a large scale environment. However, the proposal doesn't
>>> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB,
>>> which is a completely different connection from the ovs-vswitchd open-flow
>>> connection.
>>>
>>> To avoid clearing the open-flow table during ovn-controller startup, we
>>> can find a way to postpone clearing the OVS flows after the recomputing in
>>> ovn-controller is completed, right before ovn-controller replacing with the
>>> new flows.
>>>
>>> *[vi> ] *
>>>
>>> *[vi> ] Seems like we force recompute today if the OVS IDL is
>>> reconnected. Would it be possible to defer *
>>>
>>> *decision to  recompute the flows based on  the  SB’s nb_cfg we have
>>>  sync’d with? i.e.  If  our nb_cfg is *
>>>
>>> *in sync with the SB’s global nb_cfg, we can skip the recompute?  At
>>> least if nothing has changed since*
>>>
>>> *the restart, we won’t need to do anything.. We could stash nb_cfg in
>>> OVS (once ovn-controller receives*
>>>
>>> *conformation from OVS that the physical flows for an nb_cfg update are
>>> in place), which should be cleared if *
>>>
>>> *OVS itself is restarted.. (I mean currently, nb_cfg is used to check if
>>> NB, SB and Chassis are in sync, we *
>>>
>>> *could extend this to OVS/physical flows?)*
>>>
>>>
>>>
>>> *Have not thought through this though .. so maybe I am missing
>>> something…*
>>>
>>>
>>>
>>> *Thanks,*
>>>
>>>
>>>
>>> *-venu*
>>>
>>> This should largely reduce the time of connection broken during
>>> upgrading. Some changes in the ofctrl module's state machine are required,
>>> but I am not 100% sure if this approach is applicable. Need to check more
>>> details.
>>>
>>
>>
>> We can also think if its possible to do the below way
>>- When ovn-controller starts, it will not clear the flows, but instead
>> will get the dump of flows  from the br-int and populate these flows in its
>> installed flows
>> - And then when it connects to the SB DB and computes the desired
>> flows, it will anyway sync up with the installed flows with the desired
>> flows
>> - And if there is no difference between desired flows and installed
>> flows, there will be no impact on the datapath at all.
>>
>> Although this would require a careful thought and proper handling.
>>
>
> Numan, as I responded to Girish, this avoids the time spent on the
> one-time flow installation after restart (the < 10% part of the connection
> broken time), but I think currently the major problem is that > 90% of the
> time is spent on waiting for computing to finish while the OVS flows are
> already cleared. It is surely an optimization, but the most important one
> now is to avoid the 90% time. I will look at postpone clearing flows first.
>
>

I thought about this again. It seems 

Re: [ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Han Zhou]

On Fri, Aug 7, 2020 at 10:05 AM Tony Liu  wrote:

> Hi,
>
> Here are some logging snippets from ovn-controller.
> 
> 2020-08-07T16:38:04.020Z|29250|timeval|WARN|Unreasonably long 8954ms poll
> interval (8895ms user, 0ms system)
> 
> What's that mean? Is it harmless?
>
> 
> 2020-08-07T16:38:04.021Z|29251|timeval|WARN|context switches: 0 voluntary,
> 6 involuntary
> 2020-08-07T16:38:04.022Z|29252|poll_loop|INFO|wakeup due to [POLLIN] on fd
> 19 (10.6.20.91:60398<->10.6.20.86:6642) at lib/stream-fd.c:157 (99% CPU
> usage)
> 
> Is this wakeup caused by changes in sb-db?
> Why is ovn-controller so busy?
>
> 
> 2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640:
> connection dropped (Broken pipe)
> 
> Connection to local ovsdb-server is dropped.
> Is this caused by the timeout of inactivity probe?
>
> 
> 2020-08-07T16:38:04.035Z|29254|poll_loop|INFO|wakeup due to [POLLIN] on fd
> 20 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (99% CPU
> usage)
> 
> What causes this wakeup?
>
> 
> 2020-08-07T16:38:04.048Z|29255|poll_loop|INFO|wakeup due to 0-ms timeout
> at lib/ovsdb-idl.c:5391 (99% CPU usage)
> 
> What's this 0-ms wakeup mean?
>
> 
> 2020-08-07T16:38:05.022Z|29256|poll_loop|INFO|wakeup due to 962-ms timeout
> at lib/reconnect.c:643 (99% CPU usage)
> 2020-08-07T16:38:05.023Z|29257|reconnect|INFO|tcp:127.0.0.1:6640:
> connecting...
> 2020-08-07T16:38:05.041Z|29258|poll_loop|INFO|wakeup due to [POLLOUT] on
> fd 14 (127.0.0.1:51478<->127.0.0.1:6640) at lib/stream-fd.c:153 (99% CPU
> usage)
> 2020-08-07T16:38:05.041Z|29259|reconnect|INFO|tcp:127.0.0.1:6640:
> connected
> 
> Retry to connect to local ovsdb-server. A pollout event is triggered
> right after connection is established. What's poolout?
>
> ovn-controller is taking 100% CPU now, and there is no changes in
> sb-db (not busy). It seems that it's busy with local ovsdb-server
> or vswitchd. I'd like to understand why ovn-controller is so busy?
> All inactivity probe intervals are set to 30s.
>
>
Is there change from the local ovsdb? You can enable dbg log to see what is
happening.
For the local ovsdb probe, I have mentioned in the other thread that UNIX
socket is recommended (instead of tcp 127.0.0.1). Using UNIX socket
disables probe by default.

Thanks,
Han
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] [OVN] ovn-controller takes 100% cpu while no changes in sb-db

[Tony Liu]

Hi,

Here are some logging snippets from ovn-controller.

2020-08-07T16:38:04.020Z|29250|timeval|WARN|Unreasonably long 8954ms poll 
interval (8895ms user, 0ms system)

What's that mean? Is it harmless?


2020-08-07T16:38:04.021Z|29251|timeval|WARN|context switches: 0 voluntary, 6 
involuntary
2020-08-07T16:38:04.022Z|29252|poll_loop|INFO|wakeup due to [POLLIN] on fd 19 
(10.6.20.91:60398<->10.6.20.86:6642) at lib/stream-fd.c:157 (99% CPU usage)

Is this wakeup caused by changes in sb-db?
Why is ovn-controller so busy?


2020-08-07T16:38:04.022Z|29253|reconnect|WARN|tcp:127.0.0.1:6640: connection 
dropped (Broken pipe)

Connection to local ovsdb-server is dropped.
Is this caused by the timeout of inactivity probe?


2020-08-07T16:38:04.035Z|29254|poll_loop|INFO|wakeup due to [POLLIN] on fd 20 
(<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (99% CPU usage)

What causes this wakeup?


2020-08-07T16:38:04.048Z|29255|poll_loop|INFO|wakeup due to 0-ms timeout at 
lib/ovsdb-idl.c:5391 (99% CPU usage)

What's this 0-ms wakeup mean?


2020-08-07T16:38:05.022Z|29256|poll_loop|INFO|wakeup due to 962-ms timeout at 
lib/reconnect.c:643 (99% CPU usage)
2020-08-07T16:38:05.023Z|29257|reconnect|INFO|tcp:127.0.0.1:6640: connecting...
2020-08-07T16:38:05.041Z|29258|poll_loop|INFO|wakeup due to [POLLOUT] on fd 14 
(127.0.0.1:51478<->127.0.0.1:6640) at lib/stream-fd.c:153 (99% CPU usage)
2020-08-07T16:38:05.041Z|29259|reconnect|INFO|tcp:127.0.0.1:6640: connected

Retry to connect to local ovsdb-server. A pollout event is triggered
right after connection is established. What's poolout?

ovn-controller is taking 100% CPU now, and there is no changes in
sb-db (not busy). It seems that it's busy with local ovsdb-server
or vswitchd. I'd like to understand why ovn-controller is so busy?
All inactivity probe intervals are set to 30s.


Thanks!

Tony

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Double free in recent kernels after memleak fix

[Johan Knöös via discuss]

On Tue, Aug 4, 2020 at 8:52 AM Gregory Rose  wrote:
>
>
>
> On 8/3/2020 12:01 PM, Johan Knöös via discuss wrote:
> > Hi Open vSwitch contributors,
> >
> > We have found openvswitch is causing double-freeing of memory. The
> > issue was not present in kernel version 5.5.17 but is present in
> > 5.6.14 and newer kernels.
> >
> > After reverting the RCU commits below for debugging, enabling
> > slub_debug, lockdep, and KASAN, we see the warnings at the end of this
> > email in the kernel log (the last one shows the double-free). When I
> > revert 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 ("net: openvswitch:
> > fix possible memleak on destroy flow-table"), the symptoms disappear.
> > While I have a reliable way to reproduce the issue, I unfortunately
> > don't yet have a process that's amenable to sharing. Please take a
> > look.
> >
> > 189a6883dcf7 rcu: Remove kfree_call_rcu_nobatch()
> > 77a40f97030b rcu: Remove kfree_rcu() special casing and lazy-callback 
> > handling
> > e99637becb2e rcu: Add support for debug_objects debugging for kfree_rcu()
> > 0392bebebf26 rcu: Add multiple in-flight batches of kfree_rcu() work
> > 569d767087ef rcu: Make kfree_rcu() use a non-atomic ->monitor_todo
> > a35d16905efc rcu: Add basic support for kfree_rcu() batching
> >
> > Thanks,
> > Johan Knöös
>
> Let's add the author of the patch you reverted and the Linux netdev
> mailing list.
>
> - Greg

I found we also sometimes get warnings from
https://elixir.bootlin.com/linux/v5.5.17/source/kernel/rcu/tree.c#L2239
under similar conditions even on kernel 5.5.17, which I believe may be
related. However, it's much rarer and I don't have a reliable way of
reproducing it. Perhaps 50b0e61b32ee890a75b4377d5fbe770a86d6a4c1 only
increases the frequency of a pre-existing bug.

> > Traces:
> >
> > [ cut here ]
> > WARNING: CPU: 30 PID: 0 at net/openvswitch/flow_table.c:272
> > table_instance_flow_free+0x2fd/0x340 [openvswitch]
> > Modules linked in: ...
> > CPU: 30 PID: 0 Comm: swapper/30 Tainted: GE 5.6.14+ #18
> > Hardware name: ...
> > RIP: 0010:table_instance_flow_free+0x2fd/0x340 [openvswitch]
> > Code: c1 fa 1f 48 c1 e8 20 29 d0 41 39 c7 0f 8f 95 fe ff ff 48 83 c4
> > 10 48 89 ef d1 fe 5b 5d 41 5c 41 5d 41 5e 41 5f e9 33 fb ff ff <0f> 0b
> > e9 59 fe ff ff 0f 0b e8 65 f1 fe ff 85 c0 0f 85 9b fe ff ff
> > RSP: 0018:888c3e589da8 EFLAGS: 00010246
> > RAX:  RBX: 889f954ee580 RCX: dc00
> > RDX: 0007 RSI: 0003 RDI: 0246
> > RBP: 888c295150a0 R08: 9297f341 R09: 
> > R10:  R11:  R12: 889f1ed55000
> > R13: 888b72efa020 R14: 888c24209480 R15: 888b731bb6f8
> > FS:  () GS:888c3e58() knlGS:
> > CS:  0010 DS:  ES:  CR0: 80050033
> > CR2: 0733feb8a700 CR3: 000ba726e004 CR4: 003606e0
> > DR0:  DR1:  DR2: 
> > DR3:  DR6: fffe0ff0 DR7: 0400
> > Call Trace:
> > 
> > table_instance_destroy+0xf9/0x1b0 [openvswitch]
> > ? new_vport+0xb0/0xb0 [openvswitch]
> > destroy_dp_rcu+0x12/0x50 [openvswitch]
> > rcu_core+0x34d/0x9b0
> > ? rcu_all_qs+0x90/0x90
> > ? rcu_read_lock_sched_held+0xa5/0xc0
> > ? rcu_read_lock_bh_held+0xc0/0xc0
> > ? run_rebalance_domains+0x11d/0x140
> > __do_softirq+0x128/0x55c
> > irq_exit+0x101/0x110
> > smp_apic_timer_interrupt+0xfd/0x2f0
> > apic_timer_interrupt+0xf/0x20
> > 
> > RIP: 0010:cpuidle_enter_state+0xda/0x5d0
> > Code: 80 7c 24 10 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 be 04
> > 00 00 31 ff e8 c2 1a 7a ff e8 9d 4d 84 ff fb 66 0f 1f 44 00 00 <45> 85
> > ed 0f 88 b6 03 00 00 4d 63 f5 4b 8d 04 76 4e 8d 3c f5 00 00
> > RSP: 0018:888103f07d58 EFLAGS: 0246 ORIG_RAX: ff13
> > RAX:  RBX: 888c3e5c1800 RCX: dc00
> > RDX: 0007 RSI: 0006 RDI: 888103ec88d4
> > RBP: 945a3940 R08: 92982042 R09: 
> > R10:  R11:  R12: 0002
> > R13: 0002 R14: 00d0 R15: 945a3a10
> > ? lockdep_hardirqs_on+0x182/0x260
> > ? cpuidle_enter_state+0xd3/0x5d0
> > cpuidle_enter+0x3c/0x60
> > do_idle+0x36a/0x450
> > ? arch_cpu_idle_exit+0x40/0x40
> > cpu_startup_entry+0x19/0x20
> > start_secondary+0x21f/0x290
> > ? set_cpu_sibling_map+0xcb0/0xcb0
> > secondary_startup_64+0xa4/0xb0
> > irq event stamp: 1626911
> > hardirqs last  enabled at (1626910): [] 
> > __call_rcu+0x1b7/0x3b0
> > hardirqs last disabled at (1626911): []
> > trace_hardirqs_off_thunk+0x1a/0x1c
> > softirqs last  enabled at (1626882): [] 
> > irq_enter+0x75/0x80
> > softirqs last disabled at (1626883): [] 
> > irq_exit+0x101/0x110
> > ---[ end trace 8dc48dec48bb79c0 ]---
> >
> >
> > ---
>