Re: [ovs-discuss] Open V Switch module
On Fri, 3 May 2024 17:45:42 -0700 Abu Rasheda via discuss wrote: > Since the OVS kernel code has moved to the Linux kernel tree since the > 3.0 release. I did not find the module in /lib/module on Redhat 9.3, > but I did find it in > > ./modules/6.2.0-39-generic/kernel/net/openvswitch/openvswitch.ko (For > Ubuntu 22) > > Can someone guide me on how to obtain vendor (Redhat) provided OVS > kernel modules for distribution based on Linux kernel within kernel > tree OVS? The module is shipped as part of the kernel-modules-core RPM package: $ rpm -qpl ~/Downloads/kernel-modules-core*.x86_64.rpm | grep openvswitch /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch/openvswitch.ko.xz /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch/vport-geneve.ko.xz /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch/vport-gre.ko.xz /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch/vport-vxlan.ko.xz /lib/modules/5.14.0-427.13.1.el9_4.x86_64/kernel/net/openvswitch /lib/modules/5.14.0-427.13.1.el9_4.x86_64/kernel/net/openvswitch/openvswitch.ko.xz /lib/modules/5.14.0-427.13.1.el9_4.x86_64/kernel/net/openvswitch/vport-geneve.ko.xz /lib/modules/5.14.0-427.13.1.el9_4.x86_64/kernel/net/openvswitch/vport-gre.ko.xz /lib/modules/5.14.0-427.13.1.el9_4.x86_64/kernel/net/openvswitch/vport-vxlan.ko.xz HTH, fbl ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Urgent Help needed: OVS 3.2.2 Strange TC DROPs
On Wed, 17 Apr 2024 12:26:27 -0700 Gavin McKee wrote: > Hi Flavio, > > I had to restart the Open vSwitch across 16 machines to resolve the > issue for a customer . I think it will occur again and when it does > I'll use that command to gather the tc information. > > Until then I think I have found why the issue is occurring . > > Take a look at the output below (this is a packet capture from the > physical interface on the compute node , so traffic that has gone > through the OVS Openflow pipeline) - we make a 3 way handshake with R2 > , and establish the connection. A packet goes missing - TLS > handshake, it then appears that it hasn't gone through NAT as it's > using the Private IP of the VM . If that is the case, you might be able to see if the data path is matching correctly and the actions are using NAT with ovs-appctl ofproto/trace. fbl > > Take a look at frame 14 > > No. Time SourceDestination > Protocol Length Info > Delta > 14 09:24:08.064432172.27.18.244 104.18.2.35 > TLSv1502Client Hello, Alert (Level: Fatal, Description: Decode > Error) 2.362983 > > Frame 14: 502 bytes on wire (4016 bits), 502 bytes captured (4016 > bits) Ethernet II, Src: 4e:42:14:a1:2a:fb (4e:42:14:a1:2a:fb), Dst: > IETF-VRRP-VRID_ff (00:00:5e:00:01:ff) > 802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120 > Internet Protocol Version 4, Src: 172.27.18.244, Dst: 104.18.2.35 > Transmission Control Protocol, Src Port: 57394, Dst Port: 443, Seq: 1, > Ack: 1, Len: 444 > Transport Layer Security > TLSv1 Record Layer: Handshake Protocol: Client Hello > Content Type: Handshake (22) > Version: TLS 1.0 (0x0301) > Length: 432 > Handshake Protocol: Client Hello > TLSv1 Record Layer: Alert (Level: Fatal, Description: Decode > Error) Content Type: Alert (21) > Version: TLS 1.0 (0x0301) > Length: 2 > Alert Message > Level: Fatal (2) > Description: Decode Error (50) > > > > > > On Wed, 17 Apr 2024 at 11:28, Flavio Leitner wrote: > > > > > > Hi Gavin, > > > > It would be helpful if you can provide some TC dumps from the > > "good" state to the "bad" state to see how it was and what changes. > > Something like: > > > > # tc -s filter show dev enp148s0f0_1 ingress > > > > I haven't checked the attached files, but one suggestion is to > > check if this is not a csum issue. > > > > Thanks, > > fbl > > > > > > On Tue, 16 Apr 2024 13:17:10 -0700 > > Gavin McKee via discuss wrote: > > > > > Adding information relating to the Open VSwitch kernal module > > > @Ilya Maximets @Numan Siddique Can either of you help out here? > > > > > > > > > modinfo openvswitch > > > filename: > > > /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch/openvswitch.ko.xz > > > alias: net-pf-16-proto-16-family-ovs_ct_limit > > > alias: net-pf-16-proto-16-family-ovs_meter > > > alias: net-pf-16-proto-16-family-ovs_packet > > > alias: net-pf-16-proto-16-family-ovs_flow > > > alias: net-pf-16-proto-16-family-ovs_vport > > > alias: net-pf-16-proto-16-family-ovs_datapath > > > license:GPL > > > description:Open vSwitch switching datapath > > > rhelversion:9.3 > > > srcversion: 8A2159D727C8BADC82261B8 > > > depends:nf_conntrack,nf_conncount,libcrc32c,nf_nat > > > retpoline: Y > > > intree: Y > > > name: openvswitch > > > vermagic: 5.14.0-362.8.1.el9_3.x86_64 SMP preempt mod_unload > > > modversions sig_id: PKCS#7 > > > signer: Rocky kernel signing key > > > sig_key: > > > 17:CA:DE:1F:EC:D1:59:2D:9F:52:34:C6:7C:09:06:81:3D:74:7C:F7 > > > sig_hashalgo: sha256 signature: > > > 67:31:56:70:86:DB:57:69:8D:4A:9B:A7:ED:17:F3:67:65:98:97:08: > > > 1F:FB:4D:F8:A8:2D:7C:A7:7D:3A:57:85:CA:67:9D:82:72:EB:54:14: > > > F2:BB:40:78:AD:85:56:2D:EF:D5:00:95:38:A4:86:9F:5F:29:1A:81: > > > 32:94:B4:87:41:94:A0:3E:71:A5:97:44:2E:42:DD:F7:42:6B:69:94: > > > E3:AB:6E:E5:4F:C9:60:57:70:07:5F:CA:C7:83:7A:2F:C7:81:62:FF: > > > 53:AF:AC:2B:06:D8:08:D3:1D:A7:F0:43:10:98:DE:B1:62:AE:89:A5: > > > FE:EF:74:09:0F:2D:0F:D9:73:A5:59:75:D0:87:1E:EA:3A:40:86:1E: > > > 76:E5:E7:3B:59:2E:3A:7E:65:F3:92:A1:B4:84:48:3F:43:A0:D7:1C: > > > 21:29:E0:B6:D1:10:36:15:88:43:6A:11:8F:55:EE:1B:F9:53:3B:86: > > > EF:81:71:17:81:08:EC:53:30:D6:69:8E:13:11:D5:DF:15:75:88:50: > > > 69:19:51:3B:41:6B:6F:E0:7A:30:33:32:E6:60:18:02:A6:0C:63:9B: > > > C5:D7:2F:6A:D0:BA:45:03:19:0E:21:E8:18:FB:E8:D1:C1:33:05:36: > > > 1F:9B:0F:29:3F:05:51:7A:30:86:88:B7:C7:44:2E:2B:50:F9:EF:4F: > > > D4:70:EA:1B:33:E2:F0:E3:E2:88:00:E5:BF:06:E2:D4:B7:81:EE:6E: > > > 89:02:18:65:8B:1C:84:42:2F:89:14:63:1D:51:70:37:42:C5:68:DD: > > > 4D:12:7B:07:33:2B:C6:BC:8F:7F:23:D7:58:DF:47:AC:DE:08:67:FE: > > >
Re: [ovs-discuss] Urgent Help needed: OVS 3.2.2 Strange TC DROPs
Hi Gavin, It would be helpful if you can provide some TC dumps from the "good" state to the "bad" state to see how it was and what changes. Something like: # tc -s filter show dev enp148s0f0_1 ingress I haven't checked the attached files, but one suggestion is to check if this is not a csum issue. Thanks, fbl On Tue, 16 Apr 2024 13:17:10 -0700 Gavin McKee via discuss wrote: > Adding information relating to the Open VSwitch kernal module > @Ilya Maximets @Numan Siddique Can either of you help out here? > > > modinfo openvswitch > filename: > /lib/modules/5.14.0-362.8.1.el9_3.x86_64/kernel/net/openvswitch/openvswitch.ko.xz > alias: net-pf-16-proto-16-family-ovs_ct_limit > alias: net-pf-16-proto-16-family-ovs_meter > alias: net-pf-16-proto-16-family-ovs_packet > alias: net-pf-16-proto-16-family-ovs_flow > alias: net-pf-16-proto-16-family-ovs_vport > alias: net-pf-16-proto-16-family-ovs_datapath > license:GPL > description:Open vSwitch switching datapath > rhelversion:9.3 > srcversion: 8A2159D727C8BADC82261B8 > depends:nf_conntrack,nf_conncount,libcrc32c,nf_nat > retpoline: Y > intree: Y > name: openvswitch > vermagic: 5.14.0-362.8.1.el9_3.x86_64 SMP preempt mod_unload > modversions sig_id: PKCS#7 > signer: Rocky kernel signing key > sig_key: > 17:CA:DE:1F:EC:D1:59:2D:9F:52:34:C6:7C:09:06:81:3D:74:7C:F7 > sig_hashalgo: sha256 signature: > 67:31:56:70:86:DB:57:69:8D:4A:9B:A7:ED:17:F3:67:65:98:97:08: > 1F:FB:4D:F8:A8:2D:7C:A7:7D:3A:57:85:CA:67:9D:82:72:EB:54:14: > F2:BB:40:78:AD:85:56:2D:EF:D5:00:95:38:A4:86:9F:5F:29:1A:81: > 32:94:B4:87:41:94:A0:3E:71:A5:97:44:2E:42:DD:F7:42:6B:69:94: > E3:AB:6E:E5:4F:C9:60:57:70:07:5F:CA:C7:83:7A:2F:C7:81:62:FF: > 53:AF:AC:2B:06:D8:08:D3:1D:A7:F0:43:10:98:DE:B1:62:AE:89:A5: > FE:EF:74:09:0F:2D:0F:D9:73:A5:59:75:D0:87:1E:EA:3A:40:86:1E: > 76:E5:E7:3B:59:2E:3A:7E:65:F3:92:A1:B4:84:48:3F:43:A0:D7:1C: > 21:29:E0:B6:D1:10:36:15:88:43:6A:11:8F:55:EE:1B:F9:53:3B:86: > EF:81:71:17:81:08:EC:53:30:D6:69:8E:13:11:D5:DF:15:75:88:50: > 69:19:51:3B:41:6B:6F:E0:7A:30:33:32:E6:60:18:02:A6:0C:63:9B: > C5:D7:2F:6A:D0:BA:45:03:19:0E:21:E8:18:FB:E8:D1:C1:33:05:36: > 1F:9B:0F:29:3F:05:51:7A:30:86:88:B7:C7:44:2E:2B:50:F9:EF:4F: > D4:70:EA:1B:33:E2:F0:E3:E2:88:00:E5:BF:06:E2:D4:B7:81:EE:6E: > 89:02:18:65:8B:1C:84:42:2F:89:14:63:1D:51:70:37:42:C5:68:DD: > 4D:12:7B:07:33:2B:C6:BC:8F:7F:23:D7:58:DF:47:AC:DE:08:67:FE: > CB:E8:E6:4D:95:2F:6B:F5:07:4D:32:92:80:0A:7C:D1:B6:81:EE:AB: > 26:C3:C6:22:77:00:5E:64:DE:96:0E:9F:A4:A0:F0:45:9F:19:73:EB: > CC:60:AE:E9:63:E2:6D:2E:BA:65:9B:BD:04:CC:13:C2:55:88:05:03: > 1B:30:18:8B > > On Tue, 16 Apr 2024 at 11:12, Gavin McKee > wrote: > > > > Hi, > > > > I need some help with strange OVS behaviours. > > > > ovs-vsctl (Open vSwitch) 3.2.2 > > ovn-controller 23.09.1 > > Open vSwitch Library 3.2.2 > > > > TLDR: We need to restart Open VSwitch in order for TLS traffic to > > work between a VM and Cloudflare R2. After restarting Open VSwitch > > the TLS connection works fine. > > (see attached pcap tls-error.txt) > > > > See the attached openflow traces - they show a flow trace from Open > > Vswitch. > > > > Also there is a retis trace (retis tool discussed at Open VSwitch > > conference 2023). > > > > Note the drop (TC_INGRESS) in this file > > + 1702601116185568 [swapper/140] 0 [tp] skb:kfree_skb > > #60c81b6b91e2cff284fb3a3d65800 (skb 18386033671255367680) n 3 drop > > (TC_INGRESS) > > if 21 (enp148s0f0_1) rxif 21 172.27.18.244.57394 > > > 104.18.2.35.443 ttl 63 tos 0x0 id 26162 off 0 [DF] len 477 proto > > TCP (6) flags [P.] seq 792060930:792061367 ack 951229219 win 11 > > > > Again , once I restart Open vSwitch the problem goes away for a time > > and comes back sometime later (not sure what that time frame is but > > its a recurring issue.) > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss -- fbl ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVS with or without DPDK Throughput issue
On Wed, 31 Jan 2024 17:45:30 -0800
MIchael Miller via discuss wrote:
> Hi All,
>
> Not sure if this is the right forum for this question but figured I
> would start here.
>
> Have implemented several dedicated kvm based private virtualization
> server in several Outside Data Center (have no access to
> router/switch configs), Base host is Rocky Linux 9 with OVS 3.2.1
> with dpdk 22.11 complied from source and from that I have created two
> ovs bridges groups, "br-ex" is attached to 10Gbps Server NIC Card and
> vhost-net attached to a KVM running a firewall/router/vpn appliance
> ("WAN" Port), 2nd group is "br-int" which creates a form of a lan
> network without a physical NIC card, this is attached to the KVM
> running the firewall/router/vpn appliance as a "LAN" Port and any KVM
> guest as well via a vhost-net style config. When I test from Base
> Host via speedtest or iperf3, results vary but are typically
> 8+/8+Gbps and same test from KVM guest results in 4/4Gbps which
> sounds great in theory but actually throughput between KVM guests or
> KVM guest and outside world varies, have seen 31Gb take 15 minutes
> and have seen the same 31GB take several hours.
Look at the CPU usage while testing it. Most probably there are
threads scheduled over the same CPU causing the tput to go down.
See this recommendation as an example:
https://access.redhat.com/documentation/pt-br/red_hat_openstack_platform/10/html/ovs-dpdk_end_to_end_troubleshooting_guide/using_virsh_emulatorpin_in_virtual_environments_with_nfv
fbl
>
> Have gone down the rabbit hole of maybe the firewall/router/vpn
> appliance can't handle it but have tried two different vendor
> products: Sonicwall and Mikrotik CHR both have similar results. I
> have also tested a Rocky LInux KVM guest directly attached to the
> "br-ex" group but the results also show degraded performance, which
> is leading me to believe something might have been missed in the OVS
> setup.
>
>
> br-ex/br-int kvm nic config (difference is source bridge line and
> slot= line):
>
>
>
>
>
>
>
>
>
> event_idx="off" queues="8"/>
>
> function="0x0"/>
>
>
>
--
fbl
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] VxLAN in Userspace
On Tue, Jun 18, 2019 at 03:12:51PM -0400, Vasu Dasari wrote: > On Tue, Jun 18, 2019 at 12:21 PM Flavio Leitner wrote: > > > On Tue, Jun 18, 2019 at 10:41:16AM -0400, Vasu Dasari wrote: > > > Flavio, > > > > > > The device(could be a regular interface) should have been added to OVS > > with > > > "add-port" and when it is removed, ARP entries learnt on it well be > > removed > > > as well. > > > > Not necessarily. You could have a regular kernel interface with the > > tunnel endpoint address which is not part of an OVS bridge. > > > > The code being added should cover that as well. Probably you can comment on > this after you look at my code. OK. > > > The code is ready. Regarding, testing the code I am planning on extending > > > already existing case where tests are performed to check ARP entry > > > existence, like for example, "tunnel_push_pop - underlay bridge match" in > > > tunnel-push-pop.at. I hope that is fine, or should I be writing a > > separate > > > unit test case? > > > > Usually we just add more tests to the file to cover additional > > use-cases. > > > > Ok. Will add a new test case then. > > On another note, I see ARP neighbor add and show commands. This covers L3 > side. For L2 side, I see fdb show command, but there is no set command. I > would like to add that one also. Any comments? > > Current fdb show: > ovs-appctl fdb/show s1 > > Proposed CLI for fdb/set operation would look like: > ovs-appctl fdb/set > ovs-appctl fdb/set s1 2 0 00:00:01:00:00:02 > > What do you think? One could add a flow to match on specific MAC and send out on a specific port as an alternative solution which is not possible to regular bridges. On another hand, for use cases where the flow table last action is NORMAL, then having static entries might be interesting because today the FDB has only learned packets which expire/update dynamically. You also need a way to remove specific static entries because flushing the whole table has non trivial networking side effects. If you want to pursue that too, I suggest to do as a separate patchset to not mix bug fixes with enhancements. fbl ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] VxLAN in Userspace
On Tue, Jun 18, 2019 at 10:41:16AM -0400, Vasu Dasari wrote: > Flavio, > > The device(could be a regular interface) should have been added to OVS with > "add-port" and when it is removed, ARP entries learnt on it well be removed > as well. Not necessarily. You could have a regular kernel interface with the tunnel endpoint address which is not part of an OVS bridge. > The code is ready. Regarding, testing the code I am planning on extending > already existing case where tests are performed to check ARP entry > existence, like for example, "tunnel_push_pop - underlay bridge match" in > tunnel-push-pop.at. I hope that is fine, or should I be writing a separate > unit test case? Usually we just add more tests to the file to cover additional use-cases. fbl > > -Vasu > > *Vasu Dasari* > > > On Fri, Jun 14, 2019 at 9:15 AM Flavio Leitner wrote: > > > On Thu, Jun 13, 2019 at 10:58:21PM -0400, Vasu Dasari wrote: > > > Flavio, > > > > > > tnl_neigh_cache_flush(), flushes entire tunnel ARP table. When a bridge > > is > > > removed, ARP entries learnt on that bridge alone have to be removed. > > > Entries from other bridges have to be in tact. > > > > > > I am thinking of doing the following: > > > > > > 1. Write a new API in lib/tnl-neigh-cache.c: tnl_neigh_flush(const char* > > > br_name) which purges all neighbor entries on bridge br_name. > > > > That's better, indeed. > > > > > > > 2. From ofproto/ofproto-dpif-xlate.c: xlate_xbridge_remove() invoke the > > > function tnl_neigh_flush () to flush the entries. > > > > > > til_neigh_xxx() functions are getting called from ofproto-dpif-xlate.c, > > and > > > hence thinking that xlate_xbridge_remove() is a right place to do the > > job. > > > > > > What do you think? > > > > The device with the IP address can be any regular interface, so > > do you think it would work if you remove a device that is out of > > OVS? > > > > fbl > > > > > > > > -Vasu > > > > > > > > > On Thu, Jun 13, 2019 at 7:27 PM Vasu Dasari wrote: > > > > > > > Yes Flavio. Looking at code to fix this. Will send fix for review soon. > > > > > > > > Vasu > > > > > > > > > On Jun 13, 2019, at 7:15 PM, Flavio Leitner > > wrote: > > > > > > > > > >> On Wed, Jun 12, 2019 at 07:18:27PM -0400, Vasu Dasari wrote: > > > > >> Thanks Ben. Meanwhile I think I found an bug in OVS 2.9.0 with > > stale ARP > > > > >> entries after a bridge is deleted. > > > > >> > > > > >> I ran my tunneling experiment successfully. Used mininet to > > simulate the > > > > >> environment. After quitting the mininet, switch s1 is deleted. But, > > I > > > > still > > > > >> see the ARP entries in OVS. > > > > > > > > > > Looks like when route_table_valid is false we also need to > > > > > call tnl_neigh_cache_flush() otherwise you will need to wait > > > > > the ARP entry in the cache to expire (15min?) which is quite > > > > > a long time. > > > > > > > > > > Do you think you can work on a patch? > > > > > > > > > >> root@mn1:~# ovs-vsctl show > > > > >> f6af5f1c-a11c-435f-9b03-7317f364ae48 > > > > >>Manager "ptcp:6640" > > > > >>ovs_version: "2.9.0" > > > > >> > > > > >> Even thought there is no s1, I still see entries here. > > > > >> root@mn1:~# ovs-appctl tnl/arp/show > > > > >> IPMAC > > Bridge > > > > >> > > > > > > == > > > > >> 172.168.1.1 02:42:ac:14:00:02 s1 > > > > >> 172.168.1.2 02:42:ac:14:00:03 s1 > > > > >> 10.0.0.10 82:ec:29:c0:bc:ef s1 > > > > >> 10.0.0.1 d2:54:11:f0:95:df s1 > > > > >> > > > > >> > > > > >> Just for completeness, this is what I had to do to fix my > > configuration. > > > > >> > > > > >> Figured out what was wrong with my configuration. > > > > >> > > > > >> Modify my bridge s1 to be: > > > > >> > > > > >> ovs-vsctl --may-exist add-br s1 \ > > > > >>-- set Bridge s1 datapath_type=netdev \ > > > > >>-- set bridge s1 fail-mode=standalone \ > > > > >> other_config:hwaddr=$(cat /sys/class/net/eth1/address) > > > > >> > > > > >> Add the flows: > > > > >> ovs-ofctl add-flow s1 "priority=1,in_port=s1-eth1 actions=vxlan" > > > > >> ovs-ofctl add-flow s1 "priority=1,in_port=vxlan actions=s1-eth1" > > > > >> ovs-ofctl add-flow s1 "priority=0 actions=NORMAL" > > > > >> > > > > >> ip addr add 1.1.1.1/24 dev s1 > > > > >> ip link set s1 up > > > > >> ip addr flush dev eth1 2>/dev/null > > > > >> ip link set eth1 up > > > > >> > > > > >> ovs-appctl tnl/arp/set s1 1.1.1.2 00:00:01:00:00:02 > > > > > > > > > > Yeah, that will replace the old entry with the new one. > > > > > > > > > > fbl > > > > > > > > > > > > > > >> > > > > >> *Vasu Dasari* > > > > >> > > > > >> > > > > >>> On Tue, Jun 11, 2019 at 6:22 PM Ben Pfaff wrote: > > > > >>> > > > > On Tue, Jun 11, 2019 at 03:17:24PM -0400, Vasu Dasari wrote: > > >
Re: [ovs-discuss] VxLAN in Userspace
On Thu, Jun 13, 2019 at 10:58:21PM -0400, Vasu Dasari wrote: > Flavio, > > tnl_neigh_cache_flush(), flushes entire tunnel ARP table. When a bridge is > removed, ARP entries learnt on that bridge alone have to be removed. > Entries from other bridges have to be in tact. > > I am thinking of doing the following: > > 1. Write a new API in lib/tnl-neigh-cache.c: tnl_neigh_flush(const char* > br_name) which purges all neighbor entries on bridge br_name. That's better, indeed. > 2. From ofproto/ofproto-dpif-xlate.c: xlate_xbridge_remove() invoke the > function tnl_neigh_flush () to flush the entries. > > til_neigh_xxx() functions are getting called from ofproto-dpif-xlate.c, and > hence thinking that xlate_xbridge_remove() is a right place to do the job. > > What do you think? The device with the IP address can be any regular interface, so do you think it would work if you remove a device that is out of OVS? fbl > > -Vasu > > > On Thu, Jun 13, 2019 at 7:27 PM Vasu Dasari wrote: > > > Yes Flavio. Looking at code to fix this. Will send fix for review soon. > > > > Vasu > > > > > On Jun 13, 2019, at 7:15 PM, Flavio Leitner wrote: > > > > > >> On Wed, Jun 12, 2019 at 07:18:27PM -0400, Vasu Dasari wrote: > > >> Thanks Ben. Meanwhile I think I found an bug in OVS 2.9.0 with stale ARP > > >> entries after a bridge is deleted. > > >> > > >> I ran my tunneling experiment successfully. Used mininet to simulate the > > >> environment. After quitting the mininet, switch s1 is deleted. But, I > > still > > >> see the ARP entries in OVS. > > > > > > Looks like when route_table_valid is false we also need to > > > call tnl_neigh_cache_flush() otherwise you will need to wait > > > the ARP entry in the cache to expire (15min?) which is quite > > > a long time. > > > > > > Do you think you can work on a patch? > > > > > >> root@mn1:~# ovs-vsctl show > > >> f6af5f1c-a11c-435f-9b03-7317f364ae48 > > >>Manager "ptcp:6640" > > >>ovs_version: "2.9.0" > > >> > > >> Even thought there is no s1, I still see entries here. > > >> root@mn1:~# ovs-appctl tnl/arp/show > > >> IPMAC Bridge > > >> > > == > > >> 172.168.1.1 02:42:ac:14:00:02 s1 > > >> 172.168.1.2 02:42:ac:14:00:03 s1 > > >> 10.0.0.10 82:ec:29:c0:bc:ef s1 > > >> 10.0.0.1 d2:54:11:f0:95:df s1 > > >> > > >> > > >> Just for completeness, this is what I had to do to fix my configuration. > > >> > > >> Figured out what was wrong with my configuration. > > >> > > >> Modify my bridge s1 to be: > > >> > > >> ovs-vsctl --may-exist add-br s1 \ > > >>-- set Bridge s1 datapath_type=netdev \ > > >>-- set bridge s1 fail-mode=standalone \ > > >> other_config:hwaddr=$(cat /sys/class/net/eth1/address) > > >> > > >> Add the flows: > > >> ovs-ofctl add-flow s1 "priority=1,in_port=s1-eth1 actions=vxlan" > > >> ovs-ofctl add-flow s1 "priority=1,in_port=vxlan actions=s1-eth1" > > >> ovs-ofctl add-flow s1 "priority=0 actions=NORMAL" > > >> > > >> ip addr add 1.1.1.1/24 dev s1 > > >> ip link set s1 up > > >> ip addr flush dev eth1 2>/dev/null > > >> ip link set eth1 up > > >> > > >> ovs-appctl tnl/arp/set s1 1.1.1.2 00:00:01:00:00:02 > > > > > > Yeah, that will replace the old entry with the new one. > > > > > > fbl > > > > > > > > >> > > >> *Vasu Dasari* > > >> > > >> > > >>> On Tue, Jun 11, 2019 at 6:22 PM Ben Pfaff wrote: > > >>> > > On Tue, Jun 11, 2019 at 03:17:24PM -0400, Vasu Dasari wrote: > > I am running into an issue which sounds pretty basic, probably I > > might be > > missing something. > > >>> > > >>> I think you're trying to use kernel tools to configure userspace > > >>> tunnels. Did you read Documentation/howto/userspace-tunneling.rst? > > >>> > > > > > >> ___ > > >> discuss mailing list > > >> disc...@openvswitch.org > > >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] VxLAN in Userspace
On Wed, Jun 12, 2019 at 07:18:27PM -0400, Vasu Dasari wrote: > Thanks Ben. Meanwhile I think I found an bug in OVS 2.9.0 with stale ARP > entries after a bridge is deleted. > > I ran my tunneling experiment successfully. Used mininet to simulate the > environment. After quitting the mininet, switch s1 is deleted. But, I still > see the ARP entries in OVS. Looks like when route_table_valid is false we also need to call tnl_neigh_cache_flush() otherwise you will need to wait the ARP entry in the cache to expire (15min?) which is quite a long time. Do you think you can work on a patch? > root@mn1:~# ovs-vsctl show > f6af5f1c-a11c-435f-9b03-7317f364ae48 > Manager "ptcp:6640" > ovs_version: "2.9.0" > > Even thought there is no s1, I still see entries here. > root@mn1:~# ovs-appctl tnl/arp/show > IPMAC Bridge > == > 172.168.1.1 02:42:ac:14:00:02 s1 > 172.168.1.2 02:42:ac:14:00:03 s1 > 10.0.0.10 82:ec:29:c0:bc:ef s1 > 10.0.0.1 d2:54:11:f0:95:df s1 > > > Just for completeness, this is what I had to do to fix my configuration. > > Figured out what was wrong with my configuration. > > Modify my bridge s1 to be: > > ovs-vsctl --may-exist add-br s1 \ > -- set Bridge s1 datapath_type=netdev \ > -- set bridge s1 fail-mode=standalone \ > other_config:hwaddr=$(cat /sys/class/net/eth1/address) > > Add the flows: > ovs-ofctl add-flow s1 "priority=1,in_port=s1-eth1 actions=vxlan" > ovs-ofctl add-flow s1 "priority=1,in_port=vxlan actions=s1-eth1" > ovs-ofctl add-flow s1 "priority=0 actions=NORMAL" > > ip addr add 1.1.1.1/24 dev s1 > ip link set s1 up > ip addr flush dev eth1 2>/dev/null > ip link set eth1 up > > ovs-appctl tnl/arp/set s1 1.1.1.2 00:00:01:00:00:02 Yeah, that will replace the old entry with the new one. fbl > > *Vasu Dasari* > > > On Tue, Jun 11, 2019 at 6:22 PM Ben Pfaff wrote: > > > On Tue, Jun 11, 2019 at 03:17:24PM -0400, Vasu Dasari wrote: > > > I am running into an issue which sounds pretty basic, probably I might be > > > missing something. > > > > I think you're trying to use kernel tools to configure userspace > > tunnels. Did you read Documentation/howto/userspace-tunneling.rst? > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] losing packets when vxlan port is used
On Wed, Jun 12, 2019 at 02:10:19PM +0200, Miroslav Kubiczek wrote:
> Hello,
>
> I'm experiencing packets loss on OpenvSwitch host during transmission to
> vxlan port under higher traffic load.
>
> Some statistics from 2 machines are here (30 seconds run):
>
> 10.10.12.148-ens224-TX 625 100 %, 2130008249 bytes
> 10.10.12.150-ens224-RX 5999429 99 %, 2129796705 bytes
> 10.10.12.150-vxlan_sys_4789-TX 5641293 94 %, 1923680913 bytes
>
> 10.10.12.148 - is traffic generator machine
> 10.10.12.150 - is OvS host
>
> So as you can see about 5% of traffic is lost on OvS. Does anybody have a
> clue why this is happening?
> More configuration details are below.
Checkout the MTU of the interfaces because of VXLAN header size.
Also if it's under heavy load, VXLAN uses UDP and usually NIC
disables hashing on L4 header, which means a single CPU will process
all VXLAN traffic. That could be a bottleneck also.
fbl
>
> Thank you in advance for any help,
> Miroslav
>
>
> /-\ /\
> |10.0.0.2 | | |
> | ens224|-|ens224 |
> \_/ | |
> | OVS-host |
> /-\ | | /-\
> |10.0.0.3 | | vxlan_sys_4789||vxlan traffix server |
> | ens224|-|ens256 | |10.10.12.135 |
> \_/ \/ \_/
>
>
>
>
> $ sudo ovs-vsctl show
> 68e0245f-1a5f-48ec-a027-a69d839a9b11
> Bridge "br0"
> Controller "tcp:127.0.0.1:6653"
> is_connected: true
> Port "ens256"
> Interface "ens256"
> Port "br0"
> Interface "br0"
> type: internal
> Port vtep
> Interface vtep
> type: vxlan
> options: {key=flow, remote_ip="10.10.12.135"}
> Port "ens224"
> Interface "ens224"
> ovs_version: "2.5.0"
>
> $ sudo ovs-ofctl dump-flows br0 -O OpenFlow13
> OFPST_FLOW reply (OF1.3) (xid=0x2):
> cookie=0x0, duration=1265.517s, table=0, n_packets=42029479,
> n_bytes=14920464750, priority=1,in_port=10 actions=resubmit(,2)
> cookie=0x1, duration=1265.518s, table=0, n_packets=1223020,
> n_bytes=434172100, priority=1000,udp actions=resubmit(,1)
> cookie=0x0, duration=79826.502s, table=0, n_packets=1236672,
> n_bytes=439008720, priority=10,dl_dst=00:0c:29:42:63:9d actions=output:2
> cookie=0x0, duration=79794.509s, table=0, n_packets=46, n_bytes=7757,
> priority=10,dl_dst=00:0c:29:ec:54:7b actions=output:1
> cookie=0x0, duration=79826.501s, table=0, n_packets=11, n_bytes=850,
> priority=1,in_port=2,dl_dst=00:0c:29:ec:54:7b actions=output:2
> cookie=0x0, duration=79836.182s, table=0, n_packets=2, n_bytes=158,
> priority=0 actions=CONTROLLER:65535
> cookie=0x0, duration=1265.517s, table=1, n_packets=49579226,
> n_bytes=17600625230, actions=output:10
> cookie=0x0, duration=79826.501s, table=2, n_packets=42029478,
> n_bytes=14920464690, priority=100,ip,dl_dst=00:0c:29:42:63:9d
> actions=output:2
> cookie=0x0, duration=79794.509s, table=2, n_packets=0, n_bytes=0,
> priority=100,ip,dl_dst=00:0c:29:ec:54:7b actions=output:1
>
> $ ovs-vsctl --version
> ovs-vsctl (Open vSwitch) 2.5.0
> Compiled Apr 28 2016 10:25:38
> DB Schema 7.12.1
>
>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Restarting network kills ovs-vswitchd (and network)... ?
On Fri, May 17, 2019 at 09:45:36AM +, SCHAER Frederic wrote:
> Hi
>
> Thank you for your answer.
> I actually forgot to say I already had checked the syslogs, the ovs and the
> network journals/logs... no coredump reference anywhere
>
> For me a core dump or a crash would not return an exit code of 0, which seems
> to be what system saw :/
> I even straced -f the ovs-vswitchd process and made it stop/crash with an
> ifdown/ifup, but looks to me this is an exit ...
>
> (I can retry and save the strace output if necessary or usefull)
> End of strace output was (I see "brflat" in the long strings, which is the
> bridge hosting em1) :
Is this a strace of ovs-vswitchd or ovs-vsctl? Because SIGABRT
happens when the ovs-vsctl is stuck and the alarm fires. Then
this would just point that ovs-vswitchd is not running.
If ovs-vswitchd is not crashing, something is stopping the service
and maybe running sh -x /sbin/ifdown helps to shed a light?
Or add 'set -x' to /etc/sysconfig/network-scripts/if*-ovs scripts.
fbl
>
> [pid 175068] sendmsg(18, {msg_name(0)=NULL,
> msg_iov(1)=[{",\0\0\0\22\0\1\0\223\6\0\0!\353\377\377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\v\0\3\0brflat\0\0",
> 44}], msg_
> controllen=0, msg_flags=0}, 0
> [pid 175233] <... futex resumed> ) = 0
> [pid 175068] <... sendmsg resumed> )= 44
> [pid 175068] recvmsg(18,
> [pid 175234] futex(0x55b8aaa19128, FUTEX_WAKE_PRIVATE, 1
> ...skipping...
> [pid 175233] futex(0x7f7f226b9140, FUTEX_WAIT_PRIVATE, 2, NULL ...>
> [pid 175068] <... sendmsg resumed> )= 44
> [pid 175234] <... futex resumed> ) = 0
> [pid 175233] <... futex resumed> ) = -1 EAGAIN (Resource temporarily
> unavailable)
> [pid 175068] recvmsg(18,
> [pid 175234] futex(0x7f7f226b9140, FUTEX_WAIT_PRIVATE, 2, NULL ...>
> [pid 175233] futex(0x7f7f226b9140, FUTEX_WAKE_PRIVATE, 1
> [pid 175068] <... recvmsg resumed> {msg_name(0)=NULL,
> msg_iov(2)=[{"\360\4\0\0\20\0\0\0\224\6\0\0!\353\377\377\0\0\1\0\36\0\0\0C\20\1\0\0\0\0\0\v\0\3\0brflat\0\0\10\0\r\0\350\3\0\0\5\0\20\0\0\0\0\0\5\0\21\0\0\0\0\0\10\0\4\0\334\5\0\0\10\0\33\0\0\0\0\0\10\0\36\0\1\0\0\0\10\0\37\0\1\0\0\0\10\0(\0\377\377\0\0\10\0)\0\0\0\1\0\10\0
>
> \0\1\0\0\0\5\0!\0\1\0\0\0\f\0\6\0noqueue\0\10\0#\0\0\0\0\0\5\0'\0\0\0\0\0$\0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0H\234\377\377\n\0\1\0\276J\307\307\207I\0\0\n\0\2\0\377\377\377\377\377\377\0\0\304\0\27\0Y\22\5\0\0\0\0\0Uf\0\0\0\0\0\0^0j\1\0\0\0\0\372\371k\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0d\0\7\0Y\22\5\0Uf\0\0^0j\1\372\371k\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 1024},
> {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0y0\2\0\0\0\0\0\256\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\214\254\235\0\0\0\0\0
>
> z\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\211\223\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0004\0\6\0\6\0\0\0\0\0\0\0r\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0A\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\24\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\10\0\0\0\0\0",
> 65536}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 1264
> [pid 175234] <... futex resumed> ) = -1 EAGAIN (Resource temporarily
> unavailable)
> [pid 175233] <... futex resumed> ) = 0
> [pid 175234] futex(0x7f7f226b9140, FUTEX_WAKE_PRIVATE, 1
> [pid 175068] rt_sigprocmask(SIG_UNBLOCK, [ABRT],
> [pid 175234] <... futex resumed> ) = 0
> [pid 175068] <... rt_sigprocmask resumed> NULL, 8) = 0
> [pid 175068] tgkill(175068, 175068, SIGABRT
> [pid 175233] futex(0x7f7f226b9140, FUTEX_WAKE_PRIVATE, 1
> [pid 175068] <... tgkill resumed> ) = 0
> [pid 175233] <... futex resumed> ) = 0
> [pid 175068] --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=175068,
> si_uid=393} ---
> [pid 189862] +++ killed by SIGABRT +++
> [pid 175237] +++ killed by SIGABRT +++
> [pid 175236] +++ killed by SIGABRT +++
> [pid 175235] +++ killed by SIGABRT +++
> [pid 175234] +++ killed by SIGABRT +++
> [pid 175233] +++ killed by SIGABRT +++
> [pid 175232] +++ killed by SIGABRT +++
> [pid 175231] +++ killed by SIGABRT +++
> [pid 175230] +++ killed by SIGABRT +++
> [pid 175229] +++ killed by SIGABRT +++
> [pid 175228] +++ killed by SIGABRT +++
> [pid 175227] +++ killed by SIGABRT +++
> [pid 175226] +++ killed by SIGABRT +++
> [pid 175225] +++ killed by SIGABRT +++
> [pid 175224] +++ killed by SIGABRT +++
> [pid 175223] +++ killed by SIGABRT +++
> [pid 175222] +++ killed by SIGABRT +++
> [pid 175085] +++ killed by SIGABRT +++
> +++ killed by SIGABRT +++
>
Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient socket at /var/run/openvswitch
On Fri, May 17, 2019 at 06:39:26AM -0300, Flavio Leitner via discuss wrote: > On Fri, May 17, 2019 at 09:13:08AM +, Joan Vidal wrote: > > Hi Flavio, > > > > SELinux is disabled. > > Could you add the port to OVS and tell us the file ownership > and perms of the socket? (/var/run/openvswitch/vhost0) > > The same for /var/run/openvswitch directory. > > One quick test is to use /tmp instead of /var/run/openvswitch. Sorry the quick reply, I think /var/run/openvswitch is owned by root.root and then ovs cannot open the socket there. On my system: ls -lad /var/run/openvswitch/ drwxr-xr-x 2 openvswitch hugetlbfs 200 May 14 09:56 /var/run/openvswitch/ Then OVS and QEMU runs as gid hugetlbfs. fbl > > fbl > > > > > > Joan > > > > > > De: Flavio Leitner > > Enviado: viernes, 17 de mayo de 2019 10:35 > > Para: Joan Vidal > > Cc: Ian Stokes; ovs-discuss@openvswitch.org > > Asunto: Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient > > socket at /var/run/openvswitch > > > > > > Do you have SELinux enabled? Sounds like the policies are not > > updated. > > fbl > > > > On Thu, May 16, 2019 at 04:16:04PM +, Joan Vidal wrote: > > > Hi Ian, > > > > > > Upgraded QEMU version to 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.7.1) > > > Changed /etc/libvirt/qemu.conf > > > user = "root" > > > group = "root" > > > > > > And added following lines to guest XML definition: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > memAccess='shared'/> > > > > > > > > > > > > > > >> > mode='server'/> > > > > > > > > > > > > > > > > > >> > function='0x0'/> > > > > > > > > > > > >> > mode='server'/> > > > > > > > > > > > > > > > > > >> > function='0x0'/> > > > > > > > > > > > > Still getting same error: > > > > > > 2019-05-16T16:07:27.921191Z qemu-kvm: -chardev > > > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: Failed to > > > bind socket to /var/run/openvswitch/vhost0: Permission denied > > > 2019-05-16 16:07:28.140+: shutting down, reason=failed > > > > > > > > > *Joan Vidal* > > > > > > *OmniAccess* > > > > > > -Mensaje original- > > > De: Ian Stokes > > > Enviado el: 16 May 2019 16:33 > > > Para: Joan Vidal ; ovs-discuss@openvswitch.org > > > Asunto: Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient > > > socket at /var/run/openvswitch > > > > > > On 5/16/2019 3:04 PM, Joan Vidal wrote: > > > > Hi, > > > > > > > > I'm trying to use OVS-DPDK vhostuserclient ports with a qemu guest on > > > > a CentOS host. > > > > > > > > QEMU guest fails to start with the following error: > > > > > > > > error: internal error: process exited while connecting to monitor: > > > > 2019-05-16T13:15:27.481680Z qemu-kvm: -chardev > > > > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: Failed to > > > > bind socket: Permission denied 2019-05-16T13:15:27.482078Z qemu-kvm: > > > > -chardev > > > > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: chardev: > > > > opening backend "socket" failed > > > > > > > > > > > > Seems to be an issue with qemu-kvm permission to create the sockets in > > > > /var/run/openvswitch Followed this article > > > > https://www.redhat.com/en/blog/ovs-dpdk-migrating-vhostuser-socket-mod > > > > e-red-hat-openstack > > > > > > > > But is showing same error. Even running qemu-kvm and openvswitch with > > > > root is still returning Permission denied. > > > > > > > > > > > > This is the system configuration: > > > > > > > > -- > > > > Software versions: > > > > > > > > (Host
Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient socket at /var/run/openvswitch
On Fri, May 17, 2019 at 09:13:08AM +, Joan Vidal wrote:
> Hi Flavio,
>
> SELinux is disabled.
Could you add the port to OVS and tell us the file ownership
and perms of the socket? (/var/run/openvswitch/vhost0)
The same for /var/run/openvswitch directory.
One quick test is to use /tmp instead of /var/run/openvswitch.
fbl
>
>
> Joan
>
>
> De: Flavio Leitner
> Enviado: viernes, 17 de mayo de 2019 10:35
> Para: Joan Vidal
> Cc: Ian Stokes; ovs-discuss@openvswitch.org
> Asunto: Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient socket
> at /var/run/openvswitch
>
>
> Do you have SELinux enabled? Sounds like the policies are not
> updated.
> fbl
>
> On Thu, May 16, 2019 at 04:16:04PM +, Joan Vidal wrote:
> > Hi Ian,
> >
> > Upgraded QEMU version to 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.7.1)
> > Changed /etc/libvirt/qemu.conf
> > user = "root"
> > group = "root"
> >
> > And added following lines to guest XML definition:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> memAccess='shared'/>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> function='0x0'/>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> function='0x0'/>
> >
> >
> >
> > Still getting same error:
> >
> > 2019-05-16T16:07:27.921191Z qemu-kvm: -chardev
> > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: Failed to bind
> > socket to /var/run/openvswitch/vhost0: Permission denied
> > 2019-05-16 16:07:28.140+: shutting down, reason=failed
> >
> >
> > *Joan Vidal*
> >
> > *OmniAccess*
> >
> > -Mensaje original-
> > De: Ian Stokes
> > Enviado el: 16 May 2019 16:33
> > Para: Joan Vidal ; ovs-discuss@openvswitch.org
> > Asunto: Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient
> > socket at /var/run/openvswitch
> >
> > On 5/16/2019 3:04 PM, Joan Vidal wrote:
> > > Hi,
> > >
> > > I'm trying to use OVS-DPDK vhostuserclient ports with a qemu guest on
> > > a CentOS host.
> > >
> > > QEMU guest fails to start with the following error:
> > >
> > > error: internal error: process exited while connecting to monitor:
> > > 2019-05-16T13:15:27.481680Z qemu-kvm: -chardev
> > > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: Failed to
> > > bind socket: Permission denied 2019-05-16T13:15:27.482078Z qemu-kvm:
> > > -chardev
> > > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: chardev:
> > > opening backend "socket" failed
> > >
> > >
> > > Seems to be an issue with qemu-kvm permission to create the sockets in
> > > /var/run/openvswitch Followed this article
> > > https://www.redhat.com/en/blog/ovs-dpdk-migrating-vhostuser-socket-mod
> > > e-red-hat-openstack
> > >
> > > But is showing same error. Even running qemu-kvm and openvswitch with
> > > root is still returning Permission denied.
> > >
> > >
> > > This is the system configuration:
> > >
> > > --
> > > Software versions:
> > >
> > > (Host) CentOS Linux release 7.6.1810 (Core)
> > > (Guest) CentOS Linux release 7.4.1708 (Core) QEMU emulator version
> > > 1.5.3 (qemu-kvm-1.5.3-160.el7_6.1)
> >
> > The minimum QEMU version recommended for use with vhostuserclient is QEMU
> > 2.7 (See OVS Documentation below).
> >
> > http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#vhost-user-vs-vhost-user-client
> >
> > Do you see the same issue when using QEMU 2.7?
> >
> > Also from below it looks like you are using libvirt to configure QEMU?
> >
> > If so have you followed the steps outlined in the link below
> >
> > http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#adding-vhost-user-ports-to-the-guest-libvirt
> >
> > Ian
> >
> > > ovs-vswitchd (Open vSwitch) 2.11.1
> > > DPDK 18.11.0
> > >
> > >
> > > --
> > > OVS configuration
> > >
> > >
> > > #ovs-vsctl show
> > > 462375d2-8f6a-4a72-ad49-af8c00720da9
> > > Bridge br-subscriber
> > > Port br-subscriber
> > > Interface br-subscriber
> > > type: internal
> > > Port "ens1f0"
> > > Interface "ens1f0"
> > > type: dpdk
> > > options: {dpdk-devargs=":81:00.0"}
> > > Port "vhost0"
> > > Interface "vhost0"
> > > type: dpdkvhostuserclient
> > > options:
> > > {vhost-server-path="/var/run/openvswitch/vhost0"}
> > > Bridge br-internet
> > > Port br-internet
> > > Interface br-internet
> > > type: internal
> > > Port "vhost1"
> > > Interface "vhost1"
> > > type: dpdkvhostuserclient
> > > options:
> > > {vhost-server-path="/var/run/openvswitch/vhost1"}
> > > Port "ens1f1"
> > > Interface "ens1f1"
> > > type:
Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient socket at /var/run/openvswitch
Do you have SELinux enabled? Sounds like the policies are not
updated.
fbl
On Thu, May 16, 2019 at 04:16:04PM +, Joan Vidal wrote:
> Hi Ian,
>
> Upgraded QEMU version to 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.7.1)
> Changed /etc/libvirt/qemu.conf
> user = "root"
> group = "root"
>
> And added following lines to guest XML definition:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>function='0x0'/>
>
>
>
>
>
>
>
>
>
>function='0x0'/>
>
>
>
> Still getting same error:
>
> 2019-05-16T16:07:27.921191Z qemu-kvm: -chardev
> socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: Failed to bind
> socket to /var/run/openvswitch/vhost0: Permission denied
> 2019-05-16 16:07:28.140+: shutting down, reason=failed
>
>
> *Joan Vidal*
>
> *OmniAccess*
>
> -Mensaje original-
> De: Ian Stokes
> Enviado el: 16 May 2019 16:33
> Para: Joan Vidal ; ovs-discuss@openvswitch.org
> Asunto: Re: [ovs-discuss] OVS - QEMU unable to create vhostuserclient socket
> at /var/run/openvswitch
>
> On 5/16/2019 3:04 PM, Joan Vidal wrote:
> > Hi,
> >
> > I'm trying to use OVS-DPDK vhostuserclient ports with a qemu guest on
> > a CentOS host.
> >
> > QEMU guest fails to start with the following error:
> >
> > error: internal error: process exited while connecting to monitor:
> > 2019-05-16T13:15:27.481680Z qemu-kvm: -chardev
> > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: Failed to
> > bind socket: Permission denied 2019-05-16T13:15:27.482078Z qemu-kvm:
> > -chardev
> > socket,id=charnet2,path=/var/run/openvswitch/vhost0,server: chardev:
> > opening backend "socket" failed
> >
> >
> > Seems to be an issue with qemu-kvm permission to create the sockets in
> > /var/run/openvswitch Followed this article
> > https://www.redhat.com/en/blog/ovs-dpdk-migrating-vhostuser-socket-mod
> > e-red-hat-openstack
> >
> > But is showing same error. Even running qemu-kvm and openvswitch with
> > root is still returning Permission denied.
> >
> >
> > This is the system configuration:
> >
> > --
> > Software versions:
> >
> > (Host) CentOS Linux release 7.6.1810 (Core)
> > (Guest) CentOS Linux release 7.4.1708 (Core) QEMU emulator version
> > 1.5.3 (qemu-kvm-1.5.3-160.el7_6.1)
>
> The minimum QEMU version recommended for use with vhostuserclient is QEMU 2.7
> (See OVS Documentation below).
>
> http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#vhost-user-vs-vhost-user-client
>
> Do you see the same issue when using QEMU 2.7?
>
> Also from below it looks like you are using libvirt to configure QEMU?
>
> If so have you followed the steps outlined in the link below
>
> http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#adding-vhost-user-ports-to-the-guest-libvirt
>
> Ian
>
> > ovs-vswitchd (Open vSwitch) 2.11.1
> > DPDK 18.11.0
> >
> >
> > --
> > OVS configuration
> >
> >
> > #ovs-vsctl show
> > 462375d2-8f6a-4a72-ad49-af8c00720da9
> > Bridge br-subscriber
> > Port br-subscriber
> > Interface br-subscriber
> > type: internal
> > Port "ens1f0"
> > Interface "ens1f0"
> > type: dpdk
> > options: {dpdk-devargs=":81:00.0"}
> > Port "vhost0"
> > Interface "vhost0"
> > type: dpdkvhostuserclient
> > options:
> > {vhost-server-path="/var/run/openvswitch/vhost0"}
> > Bridge br-internet
> > Port br-internet
> > Interface br-internet
> > type: internal
> > Port "vhost1"
> > Interface "vhost1"
> > type: dpdkvhostuserclient
> > options:
> > {vhost-server-path="/var/run/openvswitch/vhost1"}
> > Port "ens1f1"
> > Interface "ens1f1"
> > type: dpdk
> > options: {dpdk-devargs=":81:00.1"}
> > ovs_version: "2.11.1"
> >
> > #ovs-vsctl get Open_vSwitch . other_config {dpdk-init="true",
> > dpdk-socket-mem="0,2048", pmd-cpu-mask="0x02000200"}
> >
> >
> > #cat /etc/sysconfig/openvswitch
> > OPTIONS=""
> > OVS_USER_ID="openvswitch:hugetlbfs"
> >
> >
> > # ls -la /var/run/openvswitch/
> > total 12
> > drwxrwsr-x. 3 openvswitch hugetlbfs 260 May 16 12:00 .
> > drwxr-xr-x. 29 root root 920 May 16 12:14 ..
> > srwxr-x---. 1 openvswitch hugetlbfs 0 May 16 12:00 br-internet.mgmt
> > srwxr-x---. 1 openvswitch hugetlbfs 0 May 16 12:00
> > br-internet.snoop srwxr-x---. 1 openvswitch hugetlbfs 0 May 16
> > 12:00 br-subscriber.mgmt srwxr-x---. 1 openvswitch hugetlbfs 0 May
> > 16 12:00 br-subscriber.snoop srwxr-x---. 1 openvswitch hugetlbfs 0
> > May 16 12:00 db.sock drwx--. 3 openvswitch
Re: [ovs-discuss] Restarting network kills ovs-vswitchd (and network)... ?
On Thu, May 16, 2019 at 09:34:28AM +, SCHAER Frederic wrote: > Hi, > I'm facing an issue with openvswitch, which I think is new (not even sure). > here is the description : > > * What you did that make the problem appear. > > I am configuring openstack (compute, network) nodes using OVS networks for > main interfaces and RHEL network scripts, basically using openvswitch to > create bridges, set the bridges IPs, and include the real Ethernet devices in > the bridges. > On a compute machine (not in production, so not using 3 or more interfaces), > I have for instance brflat -> em1. > Brflat has multiple IPs defined using IPADDR1, IPADDR2, etc.. > Now : at boot, machine has network. Bur if I ever change anything in network > scripts and issue either a network restart, an ifup or an ifdown : network > breaks and connectivity is lost. > > Also, on network restarts, I'm getting these logs in the network journal : > May 16 10:26:41 cloud1 ovs-vsctl[1766678]: ovs|1|vsctl|INFO|Called as > ovs-vsctl -t 10 -- --may-exist add-br brflat > May 16 10:26:51 cloud1 ovs-vsctl[1766678]: > ovs|2|fatal_signal|WARN|terminating with signal 14 (Alarm clock) > May 16 10:26:51 cloud1 network[1766482]: Bringing up interface brflat: > 2019-05-16T08:26:51Z|2|fatal_signal|WARN|terminating with signal 14 > (Alarm clock) > > * What you expected to happen. > > On network restart... to get back a working network. Not be forced to log in > using ipmi console and fix network manually. > > * What actually happened. > > What actually happens is that on ifup/ifdown/network restart, the > ovs-vswitchd daemon stops working. According to systemctl, it is actually > exiting with code 0. > If I do a ifdown on one interface, then ovs-vswitchd is down. > After ovs-vswitchd restart, I then can ifup that interface : network is still > down (no ping, nothing). > Ovs-vswitchd is again dead/stopped/exited 0. > Then : manually starting ovs-vswitchd restores connectivity. > > Please also include the following information: > * The Open vSwitch version number (as output by ovs-vswitchd --version). > ovs-vswitchd (Open vSwitch) 2.10.1 Sounds like OVS is crashing. Please check 'dmesg' if you see segmentation fault messages in there. Or the journal logs. Or the systemd service status. If it is, then the next step is to enable coredumps to grab one core. Then install openvswitch-debuginfo package to see the stack trace. You're right that ifdown should not put the service down. fbl ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd port limit with OpenStack
On Thu, May 02, 2019 at 04:44:42PM -0300, Flavio Leitner via discuss wrote: > On Tue, Apr 30, 2019 at 04:50:48PM -0700, Ben Pfaff wrote: > > On Fri, Apr 26, 2019 at 11:52:22AM -0500, William Konitzer wrote: > > > I'm reading > > > (http://www.openvswitch.org/support/dist-docs/ovs-vswitchd.8.txt > > > section LIMITS) and it says "Performance will degrade beyond 1,024 > > > ports per bridge due to fixed hash table sizing.” Do we have a little > > > more info on what that means and what to expect for less experienced > > > users like myself? > > > > I think that this comment is now obsolete. There was a fairly recent > > change that should have reduced the cost of a port. The kernel hash > > table is still fixed in size but I don't think it's accessed on any fast > > path so I think in practice it doesn't matter. > > > > > The background here is we’re working with OpenStack and seeing > > > performance issues when lots of networks are created.. Once we have > > > more than about 1500 ports on the br-int on a gateway node it seems to > > > take a long time to add new ports. > > You might want to bump the default netdev_max_backlog because that > is the maximum amount of packets queued. So, if you have too many > ports, there will be either packet loss, or slow path'ed traffic. To clarify, it depends on the actions. If you are using action NORMAL and there is a broadcast for example, all ports need a packet copy, which means more than 1k packets will be queued. IIRC OvS will slow path this case to prevent packet loss in the recent versions. fbl ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd port limit with OpenStack
On Tue, Apr 30, 2019 at 04:50:48PM -0700, Ben Pfaff wrote: > On Fri, Apr 26, 2019 at 11:52:22AM -0500, William Konitzer wrote: > > I'm reading > > (http://www.openvswitch.org/support/dist-docs/ovs-vswitchd.8.txt > > section LIMITS) and it says "Performance will degrade beyond 1,024 > > ports per bridge due to fixed hash table sizing.” Do we have a little > > more info on what that means and what to expect for less experienced > > users like myself? > > I think that this comment is now obsolete. There was a fairly recent > change that should have reduced the cost of a port. The kernel hash > table is still fixed in size but I don't think it's accessed on any fast > path so I think in practice it doesn't matter. > > > The background here is we’re working with OpenStack and seeing > > performance issues when lots of networks are created.. Once we have > > more than about 1500 ports on the br-int on a gateway node it seems to > > take a long time to add new ports. You might want to bump the default netdev_max_backlog because that is the maximum amount of packets queued. So, if you have too many ports, there will be either packet loss, or slow path'ed traffic. fbl > > > > I’m trying to quickly determine if we have a config issue, an > > Openstack issue or whether we’re hitting some sort of OVS limit as > > described. It seems to me that 1500 ports isn’t that many, but I’m not > > sure what sort of performance degradation I should be expecting above > > 1024 ports. The gateway node is so lightly loaded that I’d prefer to > > be able to handle a lot more networks on it before deploying another > > one. > > Are you adding ports one at a time with ovs-vsctl? If you can add them > in a batch, it will perform better. I guess we could also add a "daemon > mode" like ovn-nbctl, which would help a good deal too. > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
