On 6/24/22 21:50, Numan Siddique wrote: > On Fri, Jun 24, 2022 at 11:53 AM ROBERTO BARTZEN ACOSTA via discuss < > ovs-discuss@openvswitch.org> wrote: > >> Hi Dumitru, >>
Hi Roberto, >> I also think this issue is related to ovn-monitor-all=true but I'm not >> sure about the CPU usage consequences of disabling this. >> >> In OVSDB the changes are tracked and applied to each client in the IDL >> layer. The OVSDB_IDL_MONITOR is set by default, then the IDL replicates the >> changes in the database. Therefore, an OVSDB-IDL transaction modifies the >> contents of a database and the client requests information about the >> incremental changes. >> >> The ovn-monitor-all set the condition clause as true in update_sb_monitors >> function and enable the monitor for many database information, such as: >> Port_Bindings rows for local interfaces and local datapaths; Monitor >> Logical_Flow, MAC_Binding, Multicast_Group, and DNS tables for local >> datapaths; Monitor Controller_Event rows for local chassis; etc. >> >> Using these conditions clauses allows ovn-monitor-all to filter to only >> replicate when specific conditions are met. However, the default behavior >> is different, when the IDL replicates a particular table in the database, >> it replicates every row in the table. >> >> I would like to better understand the computational advantages of the >> ovn-controller conditional replication clauses, and the risks of not >> enabling this parameter in a large-scale solution. >> >> Best regards, >> Roberto >> >> > On large scale deployments, our testing has shown that - > ovn-monitor-all=false > puts a significant amount of CPU load to the Southbound ovsdb-server as it > has to conditionally send the data to each ovn-controller. > And hence we added the option - ovn-monitor-all=true. Drawback of this is > that an ovn-controller with ovn-monitor-all=true will get all the DB > updates. But this is still better compared to slow southbound ovsdb-server. > Another potential drawback of ovn-monitor-all=true is additional network traffic (all clients need to get all updates). But, as Numan mentioned above, in all our scale testing (both for OpenStack and OpenShift) the impact of this seems to be less significant compared to the impact on the Southbound when ovn-monitor-all=false. Numan, to avoid using ovn-monitor-all=false as workaround in this case, do you think we can port https://github.com/ovn-org/ovn/commit/0a4e073f4124b58f1b21778ec2293bbc4180e3e0 to branch-21.09 and see if Xena can pick it up? Thanks, Dumitru > Thanks > Numan > > > >> Em sex., 24 de jun. de 2022 às 12:13, Tiago Pires <tiag...@gmail.com> >> escreveu: >> >>> Hi Dumitru, >>> >>> I did a test and configuring ovn-monitor-all as false to solve this >>> behaviour. >>> It seems the option I have now is to use it as a workaround until I have >>> conditions to upgrade to Yoga that has OVN 22.03. >>> >>> Thank you for your help. >>> >>> Regards, >>> >>> Tiago Pires >>> >>> >>> Em sex., 24 de jun. de 2022 às 04:27, Dumitru Ceara <dce...@redhat.com> >>> escreveu: >>> >>>> On 6/23/22 22:23, Tiago Pires wrote: >>>>> Hi all, >>>>> >>>> >>>> Hi Tiago, >>>> >>>>> I did some troubleshooting and I'm seeing this error (ovs-vswitchd) >>>> always >>>>> when a VM is created in a Chassi: >>>>> 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network >>>> device >>>>> tap8a43df0c-fd (No such device) >>>>> 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added >>>> interface >>>>> tap8a43df0c-fd on port 51 >>>>> 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added >>>> interface >>>>> tap3200bf1c-20 on port 52 >>>>> 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430 >>>>> flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes) >>>>> >>>> >>>> It doesn't look to me like there's anything to worry about from these >>>> logs. >>>> >>>>> On this commit >>>>> >>>> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dce...@redhat.com/ >>>>> it solved something similar to my issue. It seems the ovs-vswitchd is >>>>> missing some flows and when I run the recompute it fixes it. >>>>> So, to avoid this issue I'm testing at this moment to run the recompute >>>>> through libvirt hook when a VM gets "started" status. >>>>> >>>> >>>> While this might "fix" the issue it's not really ideal. ovn-controller >>>> should properly install the flows all the time. Otherwise it's a bug. >>>> >>>>> Regards, >>>>> >>>>> Tiago Pires >>>>> >>>>> >>>>> Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires <tiag...@gmail.com> >>>>> escreveu: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I'm trying to understand a stranger's behaviour regarding to >>>>>> ovn-controller. >>>>>> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes when a >>>> new >>>>>> VM is created, this VM can reach other VMs in east-west traffic (even >>>> in >>>>>> differents Chassis) but it can't reach an external network (e.g. >>>> Internet) >>>>>> through Chassi Gateway. >>>>>> I ran the following trace: >>>>>> # ovs-appctl ofproto/trace br-int >>>>>> >>>> in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64 >>>>>> >>>>>> And I got this output: >>>>>> Final flow: >>>>>> >>>> recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x0000,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 >>>>>> Megaflow: >>>>>> >>>> recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src= >>>>>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no >>>>>> Datapath actions: >>>>>> >>>> ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535)) >>>>>> It seems the Datapath is querying the controller and I did not >>>> understand >>>>>> the reason. >>>>>> >>>>>> So, I did an ovn-controller recompute (ovn-appctl -t ovn-controller >>>>>> recompute) on the Chassi where the VM is placed to check if it could >>>> change >>>>>> the behaviour and I could trace the packet with success and the VM >>>> started >>>>>> to communicate with the Internet normally: >>>>>> Final flow: >>>>>> >>>> recirc_id=0x2,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x0000,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 >>>>>> Megaflow: >>>>>> >>>> recirc_id=0x2,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,tun_id=0/0xffffff,tun_metadata0=NP,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src= >>>>>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ecn=0,nw_ttl=64,nw_frag=no >>>>>> Datapath actions: >>>>>> >>>> ct(commit,zone=15,label=0/0x1,nat(src)),set(tunnel(tun_id=0x2a,dst=10.X6.X3.133,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30002}),flags(df|csum|key))),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:5e:00:04:00)),set(ipv4(ttl=63)),2 >>>>>> The Datapath action is using the tunnel with the Chassi Gateway. >>>>>> >>>>>> It happens always with new VMs but sometimes. After running the >>>> recompute >>>>>> on the Chassi, I created additional VMs and this issue did not happen. >>>>>> >>>>>> In my Chassi I have enable these parameters also: >>>>>> ovn-monitor-all="true" >>>>>> ovn-openflow-probe-interval="0" >>>>>> ovn-remote-probe-interval="180000" >>>>>> >>>>>> Do you know this behaviour could be bug related? >>>> >>>> This is most definitely a bug. >>>> >>>> Very likely it's the bug that was fixed in this commit: >>>> >>>> https://github.com/ovn-org/ovn/commit/0a4e073f4124b58f1b21778ec2293bbc4180e3e0 >>>> >>>> The change is available in the 21.12 stable branch and later. So you >>>> need to upgrade the OVN version in your OpenStack deployment to >>>> something that includes it. >>>> >>>> Hope this helps. >>>> >>>> Regards, >>>> Dumitru >>>> >>>>>> >>>>>> Tiago Pires >>>>>> >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list >>>>> disc...@openvswitch.org >>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>>> >>>> _______________________________________________ >>> discuss mailing list >>> disc...@openvswitch.org >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>> >> >> >> >> *‘Esta mensagem é direcionada apenas para os endereços constantes no >> cabeçalho inicial. Se você não está listado nos endereços constantes no >> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >> mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão >> imediatamente anuladas e proibidas’.* >> >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para >> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não >> poderá aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos’.* >> _______________________________________________ >> discuss mailing list >> disc...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
