Re: [ovs-discuss] OVN: Delay in handling unixctl commands in ovsdb-server
On Wed, Feb 12, 2020 at 9:57 AM Numan Siddique wrote: > > Hi Ben/All, > > In an OVN deployment - with OVN dbs deployed as active/standby using > pacemaker, we are seeing delays in response to unixctl command - > ovsdb-server/sync-status. > > Pacemaker periodically calls the OVN pacemaker OCF script to get the > status and this script internally invokes - ovs-appctl -t > /var/run/openvswitch/ovnsb_db.ctl ovsdb-server/sync-status. In a large > deployment with lots of OVN resources we see that ovsdb-server takes a > lot of time (sometimes > 60 seconds) to respond to this command. This > causes pacemaker to stop the service in that node and move the master > to another node. This causes a lot of disruption. > > One approach of solving this issue is to handle unixctl commands in a > separate thread. The commands like sync-status, get-** etc can be > easily handled in the thread. Still, there are many commands like > ovsdb-server/set-active-ovsdb-server, ovsdb-server/compact etc (which > changes the state) which needs to be synchronized between the main > ovsdb-server thread and the newly added thread using a mutex. > > Does this approach makes sense ? I started working on it. But I wanted > to check with the community before putting into more efforts. > > Are there better ways to solve this issue ? > > Thanks > Numan > Hi Numan, It seems reasonable to me. Multi-threading would add a little complexity, but in this case it should be straightforward. It merely requires mutexes to synchronize between the threads for *writes*, and also for *reads* of non-atomic data. The only side effect is that *if* the thread that does the DB job really stucked because of a bug and not handling jobs at all, the unixctl thread ovsdb-server/sync-status command wouldn't detect it, so it could result in pacemaker reporting *happy* status without detecting problems. First for all this is unlikely to happen. But if we really think it is a problem we can still solve it by incrementing a counter in main loop and have a new command (readonly, without mutex) to check if this counter is increasing, to tell if the server if really working. Thanks, Han ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] OVN solid underlay
Hi, we are currently using the neutron reference setup in our Openstack platform (python agents + OVS). Our "underlay" is a layer 3 CLOS (OSPF Cisco) with L2 VLANs from Leaf to Servers and subnets per rack (all IPv4). But we would like to migrate to OVN and add Ironic to our service portfolio. Are there any recommendations regarding OVN and a solid underlay that still supports Ironic? We have to exchange the switches anyway (EOL) so we are open for any new vendor. What about cumulus? All the best, Florian smime.p7s Description: S/MIME cryptographic signature ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] OVN: Delay in handling unixctl commands in ovsdb-server
Hi Ben/All, In an OVN deployment - with OVN dbs deployed as active/standby using pacemaker, we are seeing delays in response to unixctl command - ovsdb-server/sync-status. Pacemaker periodically calls the OVN pacemaker OCF script to get the status and this script internally invokes - ovs-appctl -t /var/run/openvswitch/ovnsb_db.ctl ovsdb-server/sync-status. In a large deployment with lots of OVN resources we see that ovsdb-server takes a lot of time (sometimes > 60 seconds) to respond to this command. This causes pacemaker to stop the service in that node and move the master to another node. This causes a lot of disruption. One approach of solving this issue is to handle unixctl commands in a separate thread. The commands like sync-status, get-** etc can be easily handled in the thread. Still, there are many commands like ovsdb-server/set-active-ovsdb-server, ovsdb-server/compact etc (which changes the state) which needs to be synchronized between the main ovsdb-server thread and the newly added thread using a mutex. Does this approach makes sense ? I started working on it. But I wanted to check with the community before putting into more efforts. Are there better ways to solve this issue ? Thanks Numan ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] NetworkManager - openvswitch - CentOS8 KVM
Hello Guys, I am trying to create a network in KVM using openvswitch. network-scripts are deprecated CentOS8. So need to use NetworkManager with openvswitch. I have created ovs bridge like below. == [root@c8kvm ~]# nmcli conn add con-name ovs-br1 type ovs-bridge conn.interface ovs-br1 [root@c8kvm ~]# nmcli conn add con-name ovs-br1-port0 type ovs-port conn.interface ovs-br1-port0 master ovs-br1 [root@c8kvm ~]# nmcli conn add con-name ovs-br1-ens4 type ovs-interface slave-type ovs-port conn.interface ovs-br1-ens4 master ovs-br1-port0 ipv4.method manual ipv4.address 192.168.0.10/24 [root@c8kvm ~]# nmcli conn add con-name ovs-br1-port1 type ovs-port conn.interface ovs-br1-port1 master ovs-br1 [root@c8kvm ~]# nmcli conn add con-name ens4 type ethernet conn.interface ens4 master ovs-br1-port1 [root@c8kvm ~]# ovs-vsctl show ddfe4286-c080-431b-8cc7-14acb756333c Bridge "ovs-br1" Port "ovs-br1-port1" Interface "ens4" type: system Port "ovs-br1-port0" Interface "ovs-br1-ens4" type: internal ovs_version: "2.12.0" [root@c8kvm ~]# nmcli c s NAME UUID TYPE DEVICE ovs-br1-ens4 719839f8-02fe-46f8-a68d-2157e54fb778 ovs-interface ovs-br1-ens4 ens4 f6a7103c-e5c3-41c4-9a45-9ecca4a1aad4 ethernet ens4 ovs-br1a496b994-1b78-4045-98e7-dfaffd79d54e ovs-bridge ovs-br1 ovs-br1-port0 9f47f7e6-b3b1-44a4-b492-afede0c16964 ovs-port ovs-br1-port0 ovs-br1-port1 be3caec2-9bd0-4c40-b4f2-c4b7537e2cb9 ovs-port ovs-br1-port1 == After this, I have defined a KVM network. = [root@c8kvm ~]# virsh net-list --all Name State Autostart Persistent -- ovs-private active yes yes [root@c8kvm ~]# virsh net-dumpxml ovs-private ovs-private aa933120-6e67-4a86-bfdf-e87709da0176 = The problem is when trying to create a VM using this network getting the following error. [root@c8kvm ~]# virt-install --name test-c8 --vcpus 2 --memory 2048 --autostart --disk path=/var/lib/libvirt/images/test-c8.qcow2 --os-variant centos7.0 --network network:ovs-private --graphics vnc,listen=0.0.0.0,port=5999,password='hello123' --console pty,target_type=serial --cdrom /var/lib/libvirt/images/CentOS-8-x86_64-1905-dvd1.iso WARNING Unable to connect to graphical console: virt-viewer not installed. Please install the 'virt-viewer' package. WARNING No console to launch for the guest, defaulting to --wait -1 Starting install... ERRORCannot get interface MTU on 'ovs-br1': No such device Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start test-c8 otherwise, please restart your installation. While inspecting /var/log/messages, can see errors. Feb 12 04:03:35 c8kvm NetworkManager[24403]: [1581498215.5522] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/10) Feb 12 04:03:35 c8kvm systemd-udevd[24953]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Feb 12 04:03:35 c8kvm libvirtd[13725]: 2020-02-12 09:03:35.557+: 13727: error : virNetDevGetMTU:430 : Cannot get interface MTU on 'ovs-br1': No such device Feb 12 04:03:35 c8kvm libvirtd[13725]: 2020-02-12 09:03:35.569+: 13747: error : virFileReadAll:1460 : Failed to open file '/sys/class/net/vnet0/operstate': No such file or directory Feb 12 04:03:35 c8kvm libvirtd[13725]: 2020-02-12 09:03:35.569+: 13747: error : virNetDevGetLinkInfo:2552 : unable to read: /sys/class/net/vnet0/operstate: No such file or directory Feb 12 04:03:35 c8kvm ovs-vsctl[24962]: ovs|1|db_ctl_base|ERR|'del-port' command requires at least 1 arguments Feb 12 04:03:35 c8kvm libvirtd[13725]: 2020-02-12 09:03:35.629+: 13727: error : virCommandWait:2678 : internal error: Child process (ovs-vsctl --timeout=5 -- --if-exists del-port) unexpected exit status 1: ovs-vsctl: 'del-port' command requires at least 1 arguments Feb 12 04:03:35 c8kvm libvirtd[13725]: 2020-02-12 09:03:35.629+: 13727: error : virNetDevOpenvswitchRemovePort:231 : internal error: Unable to delete port (null) from OVS Is there any way to fix this? *Best Regards,* *Arvin* ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
