Hi List, I am currently facing an issue where OVS is crashing occasionally with the following error message.
|util(handler59)|EMER|../include/openvswitch/ofpbuf.h:196: assertion offset + size <= b->size failed in ofpbuf_at_assert() Or util(revalidator151)|EMER|../include/openvswitch/ofpbuf.h:196: assertion offset + size <= b->size failed in ofpbuf_at_assert() Once this is shown in the log, the ovs container restarts and continues working for a while. It looks similar to [ovs-discuss] ovs-vswitchd.service crashes (openvswitch.org)<https://mail.openvswitch.org/pipermail/ovs-discuss/2019-November/049464.html> but no real resolution was reached other than to disable hardware offload. Does anyone have any idea how we can go about troubleshooting and hopefully resolving without disabling hardware offload, please let me know if you need anything else, and I appreciate people taking the time to look. Environment and required outputs are below. Melanox Connect X 5 OpenStack Zed Linux Kernel: Linux version 5.19.0-41-generic Linux Distro: Ubuntu 22.04 ovs-vswitchd (Open vSwitch) 3.0.3 ovs-dpctl show system@ovs-system: lookups: hit:42366200 missed:183200724 lost:145155 flows: 5 masks: hit:521980916 total:3 hit/pkt:2.31 cache: hit:40602017 hit-rate:18.00% caches: masks-cache: size:256 port 0: ovs-system (internal) port 1: br-ex (internal) port 2: bond0 port 3: br-int (internal) port 4: genev_sys_6081 (geneve: packet_type=ptap) port 5: tap28040ca3-fc port 6: tap8c92c51e-86 port 7: tap51552232-63 port 8: tap4597d793-96 port 9: tap64824b66-b0 port 10: tap44156c60-e0 port 11: tapccc3f7df-60 port 12: tap91794e1d-22 port 13: tap3a88326e-b0 port 14: tap6f3f61b7-02 port 15: tap53ffc01b-ba port 16: tapdd39a613-5c port 17: tap968fdb09-e0 port 25: tap7e3dba4a-e9 port 31: eth15 port 39: tapd6ca95fe-14 port 40: tap3ff400db-42 port 42: tap09588180-22 port 43: tapb140af49-cc port 44: tapdb3e7d56-de port 45: tap8eae39a4-51 ovs-ofctl show br-int OFPT_FEATURES_REPLY (xid=0x2): dpid:000086032b270643 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 16(ovn-ock000-7): addr:26:4a:b1:e3:dc:04 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 19(ovn-ock000-2): addr:3a:3a:a0:62:e9:51 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 257(ovn-ock000-a): addr:6e:f9:fb:a2:25:a7 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 258(ovn-ock000-e): addr:92:31:4d:10:2c:5f config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 260(ovn-occ000-0): addr:fa:7e:e5:67:45:46 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 261(ovn-ock000-11): addr:4e:21:16:51:92:b1 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 262(ovn-ock000-f): addr:72:eb:19:24:8a:3e config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 263(ovn-occ000-1): addr:fe:ae:53:f1:fa:fb config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 264(ovn-occ000-2): addr:22:33:2c:bc:84:ee config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 269(patch-br-int-to): addr:a2:ec:6d:1e:dc:16 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 271(patch-br-int-to): addr:12:9d:45:7c:c1:e6 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 273(patch-br-int-to): addr:fe:03:c1:dc:e2:69 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 274(patch-br-int-to): addr:66:2c:6a:50:14:14 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 275(patch-br-int-to): addr:1e:a1:66:89:1d:98 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 276(patch-br-int-to): addr:f2:dd:26:31:7e:ff config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 277(patch-br-int-to): addr:9a:5e:1c:a6:fc:27 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 296(eth15): addr:86:88:3e:bb:0a:5f config: 0 state: 0 current: AUTO_NEG advertised: 1GB-FD 10GB-FD AUTO_NEG AUTO_PAUSE supported: 1GB-FD 10GB-FD AUTO_NEG AUTO_PAUSE speed: 0 Mbps now, 10000 Mbps max 299(patch-br-int-to): addr:f2:be:71:df:24:fe config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 345(tapd6ca95fe-14): addr:fe:16:3e:4f:87:da config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 346(tap3ff400db-42): addr:fe:16:3e:f0:6c:ce config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 348(tap09588180-22): addr:fe:16:3e:a1:8e:fb config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 349(tapb140af49-cc): addr:fe:16:3e:ce:8b:66 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 350(tapdb3e7d56-de): addr:fe:16:3e:0c:b8:71 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 351(tap8eae39a4-51): addr:da:92:44:fe:34:54 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 538(ovn-ock000-0): addr:32:46:ca:2b:34:c1 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 539(tap28040ca3-fc): addr:fe:16:3e:4f:28:ed config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 548(tap91794e1d-22): addr:fe:16:3e:e9:cc:a0 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 575(tap64824b66-b0): addr:62:85:6d:46:cc:7a config: 0 state: 0 current: 10GB-FD COPPER speed: 10000 Mbps now, 0 Mbps max 576(tap44156c60-e0): addr:3a:2b:82:c3:5c:34 config: 0 state: 0 current: 10GB-FD COPPER speed: 10000 Mbps now, 0 Mbps max 577(tapccc3f7df-60): addr:ee:1c:70:2e:ae:b0 config: 0 state: 0 current: 10GB-FD COPPER speed: 10000 Mbps now, 0 Mbps max 578(tap3a88326e-b0): addr:ca:c6:a9:e6:2f:bb config: 0 state: 0 current: 10GB-FD COPPER speed: 10000 Mbps now, 0 Mbps max 604(tap51552232-63): addr:fe:16:3e:66:77:12 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 605(tap968fdb09-e0): addr:ca:ea:cf:d1:1e:5f config: 0 state: 0 current: 10GB-FD COPPER speed: 10000 Mbps now, 0 Mbps max 636(tapdd39a613-5c): addr:fe:16:3e:71:21:f4 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 640(tap7e3dba4a-e9): addr:fe:16:3e:72:b6:b4 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 641(tap8c92c51e-86): addr:fe:16:3e:dd:e0:9f config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 642(tap4597d793-96): addr:fe:16:3e:1a:46:5d config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 643(patch-br-int-to): addr:3e:ee:8d:25:39:61 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 644(tap6f3f61b7-02): addr:1e:92:09:65:32:1c config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 645(tap53ffc01b-ba): addr:fe:16:3e:23:1a:74 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max 655(patch-br-int-to): addr:f2:c3:ba:ef:73:ba config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 689(patch-br-int-to): addr:46:03:36:f2:68:b7 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 690(patch-br-int-to): addr:8a:2d:d1:90:38:90 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(br-int): addr:86:03:2b:27:06:43 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 ovs-ofctl show br-ex OFPT_FEATURES_REPLY (xid=0x2): dpid:00000c42a15e3edc n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 1(bond0): addr:0c:42:a1:5e:3e:dc config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 2(patch-provnet-b): addr:7e:5f:0f:65:8c:99 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 5(patch-provnet-1): addr:0a:09:ab:34:2f:1a config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 6(patch-provnet-8): addr:6e:22:eb:ce:bc:70 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 7(patch-provnet-8): addr:46:13:4b:d8:f5:04 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 8(patch-provnet-7): addr:62:c8:c7:ac:67:f2 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 11(patch-provnet-5): addr:76:ed:b6:15:59:06 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 12(patch-provnet-a): addr:f2:18:ef:db:88:95 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 13(patch-provnet-0): addr:0e:cf:32:03:e4:0e config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 15(patch-provnet-5): addr:da:6f:86:4c:c5:a3 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 16(patch-provnet-f): addr:ee:46:07:bd:63:2c config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 17(patch-provnet-3): addr:26:e3:68:3d:e2:07 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 18(patch-provnet-0): addr:0a:6d:46:e2:f8:45 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(br-ex): addr:0c:42:a1:5e:3e:dc config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 Rgds Steve Relf Head of Cloud Engineering EscherCloud AI "The future has arrived - it's just not evenly distributed yet" - William Gibson. ________________________________ EscherCloudAI BV | EscherCloudAI UK Ltd | www.eschercloud.ai The information contained in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. If you are not the intended recipient you are hereby (a): notified that any disclosure, copying, distribution or taking any action with respect to the content of this information is strictly prohibited and may be unlawful, and (b): kindly requested to inform the sender immediately and destroy any copies. EschercloudAI does neither accept any responsibility and / or liability for the improper and incomplete transmission of the information contained in this communication nor for any delay in its receipt. ________________________________
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
