On Fri, Dec 23, 2022 at 10:27 AM Gavin McKee <gavmcke...@googlemail.com> wrote: > > Frode, > > Thanks for the great information . What are your thoughts regarding having > an ability to view the port_table from an ovn-appctl command ? I think this > would be helpful to troubleshoot issues when plugging / unplugging > representor devices from the integration bridge ?
Yep, that is a good idea. To get you unblocked you can try these two patches: https://pastebin.ubuntu.com/p/SHf8QZYsRC/ https://pastebin.ubuntu.com/p/kKwPyHtTfR/ Note that this might not work well with bonding for the non-DPU case, and we need to figure something out there. There is a difference between the DPU and non-DPU use cases wrt. how/where bonding is set up. For the DPU use case the bonding is set up at the DPU side, and the host only sees one PF. For the non-DPU use case you typically set up bonding on the host and firmware features such as VF-LAG ensure connectivity for VFs on both PFs regardless of link status. A side effect of having the bond config on the host is that the MAC address of the PFs is changed so that the bond and both PF interfaces share the same MAC address. This of course does not work well with the current data model for lookup. -- Frode Nordahl > Gav > > On Thu, 22 Dec 2022 at 19:46, Frode Nordahl <frode.nord...@canonical.com> > wrote: >> >> On Wed, Dec 21, 2022 at 4:08 PM Gavin McKee <gavmcke...@googlemail.com> >> wrote: >> > >> > Hi Frode, >> > >> > Thanks for your email and for taking the time to look at this. >> >> You're very welcome, thank you for collecting more information. >> >> > Kernel version >> > >> > root@usc01a-032-16a:/home/gmckee# uname -r >> > 5.15.0-53-generic >> >> At that kernel version the `hw_addr` attribute is available, however >> as discussed below this does not help us for the non-DPU use case. >> >> > Output from the devlink port does not show a hw_addr on the physical port . >> > >> > root@usc01a-032-16a:/home/gmckee# devlink port >> > pci/0000:07:00.0/65535: type eth netdev enp7s0f0 flavour physical port 0 >> > splittable false >> > pci/0000:07:00.0/1: type eth netdev enp7s0f0_0 flavour pcivf controller 0 >> > pfnum 0 vfnum 0 external false splittable false >> > function: >> > hw_addr 10:70:fd:ab:cd:01 >> > pci/0000:07:00.0/2: type eth netdev enp7s0f0_1 flavour pcivf controller 0 >> > pfnum 0 vfnum 1 external false splittable false >> > function: >> > hw_addr 10:70:fd:ab:cd:02 >> > pci/0000:07:00.0/3: type eth netdev enp7s0f0_2 flavour pcivf controller 0 >> > pfnum 0 vfnum 2 external false splittable false >> > function: >> > hw_addr 10:70:fd:ab:cd:03 >> > pci/0000:07:00.0/4: type eth netdev enp7s0f0_3 flavour pcivf controller 0 >> > pfnum 0 vfnum 3 external false splittable false >> > function: >> > hw_addr 10:70:fd:ab:cd:04 >> > pci/0000:07:00.0/5: type eth netdev enp7s0f0_4 flavour pcivf controller 0 >> > pfnum 0 vfnum 4 external false splittable false >> > function: >> > hw_addr 10:70:fd:e2:44:44 >> > pci/0000:07:00.0/6: type eth netdev enp7s0f0_5 flavour pcivf controller 0 >> > pfnum 0 vfnum 5 external false splittable false >> > function: >> > hw_addr 10:70:fd:e2:a3:02 >> > pci/0000:07:00.1/131071: type eth netdev enp7s0f1 flavour physical port 1 >> > splittable false >> > pci/0000:07:00.3/196608: type eth netdev enp7s0f0v1 flavour virtual >> > splittable false >> > pci/0000:07:00.4/262144: type eth netdev enp7s0f0v2 flavour virtual >> > splittable false >> > >> > Log messages below >> > >> > root@usc01a-032-16a:/home/gmckee# ovn-controller >> > 2022-12-20T21:42:02Z|00001|vif_plug_representor|WARN|attempt to add >> > function before having knowledge about PF >> >> [ snip ] >> >> The representor plugin currently assumes the presence of PCI_PF >> flavoured ports and uses them to correlate to which PF each PCI_VF >> port belongs. When the ovn-controller runs on the SmartNIC DPU side of >> a PCI complex, these ports represent resources presented to the host >> side of the PCI complex. >> >> When using an accelerator card that exposes the physical ports and the >> embedded switch to the host system there will be no PCI_PF ports. The >> devlink-port infrastructure is still useful in this topology because >> it can provide a unified way of managing subfunctions [2]. >> >> I guess we could try to find a way to detect this mode of operation, >> or introduce an option, and then use some other method to correlate >> the resources. >> >> At the point in time where the message is logged [3], there is a lot >> of information available, so we just need to find a light weight and >> compatible way of doing it. >> >> 2: >> https://legacy.netdevconf.info/0x14/pub/papers/45/0x14-paper45-talk-paper.pdf >> 3: >> https://github.com/ovn-org/ovn-vif/blob/ce1a36f300a74b4eae55a7fec7d18da8b9218e29/lib/vif-plug-providers/representor/vif-plug-representor.c#L321-L328 >> >> -- >> Frode Nordahl >> >> > On Wed, 21 Dec 2022 at 02:50, Frode Nordahl <frode.nord...@canonical.com> >> > wrote: >> >> >> >> Hello, Gavin, >> >> >> >> Thank you for your interest in the vif plug infrastructure and the >> >> representor port plugin. See replies inline below. >> >> >> >> tir. 20. des. 2022, 19:29 skrev Gavin McKee via discuss >> >> <ovs-discuss@openvswitch.org>: >> >>> >> >>> Hi, >> >>> >> >>> We are hoping someone can help with the following error message. >> >>> >> >>> Here we add the required options to the logical switch port in OVN North >> >>> ovn-nbctl lsp-set-options c1-sw0-p1 requested-chassis=usc01a-032-16a >> >>> vif-plug-type=representor vif-plug:representor:pf-mac=10:70:fd:df:9c:3a >> >>> vif-plug:representor:vf-num=4 >> >>> >> >>> When I check the ovn-controller log on the hypervisor I see the >> >>> following error message: >> >>> 2022-12-20T18:24:42.815Z|00108|vif_plug|INFO|Not plugging lport >> >>> c1-sw0-p1 on direction from VIF plug provider. >> >>> 2022-12-20T18:24:47.816Z|00109|vif_plug_representor|INFO|No representor >> >>> port found for lport: c1-sw0-p1 pf-mac: '10:70:fd:df:9c:3a' vf-num: '4' >> >>> >> >>> Here is the information for the Mellanox Connect X6 card we are using , >> >>> you can see the mac on the physical interface is defined in the entry >> >>> vif-plug:representor:pf-mac=10:70:fd:df:9c:3a >> >>> ``` >> >>> root@usc01a-032-16a:/home/gmckee# ip link show enp7s0f0 >> >>> 14: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc mq master >> >>> ovs-system state UP mode DEFAULT group default qlen 1000 >> >>> link/ether 10:70:fd:df:9c:3a brd ff:ff:ff:ff:ff:ff >> >>> vf 0 link/ether 10:70:fd:ab:cd:01 brd ff:ff:ff:ff:ff:ff, spoof >> >>> checking off, link-state disable, trust off, query_rss off >> >>> vf 1 link/ether 10:70:fd:ab:cd:02 brd ff:ff:ff:ff:ff:ff, spoof >> >>> checking off, link-state disable, trust off, query_rss off >> >>> vf 2 link/ether 10:70:fd:ab:cd:03 brd ff:ff:ff:ff:ff:ff, spoof >> >>> checking off, link-state disable, trust off, query_rss off >> >>> vf 3 link/ether 10:70:fd:ab:cd:04 brd ff:ff:ff:ff:ff:ff, spoof >> >>> checking off, link-state disable, trust off, query_rss off >> >>> vf 4 link/ether 10:70:fd:e2:44:44 brd ff:ff:ff:ff:ff:ff, spoof >> >>> checking off, link-state disable, trust off, query_rss off >> >>> vf 5 link/ether 10:70:fd:e2:a3:02 brd ff:ff:ff:ff:ff:ff, spoof >> >>> checking off, link-state disable, trust off, query_rss off >> >>> altname enp7s0f0np0 >> >>> ``` >> >>> >> >>> The representor information is as follows >> >>> >> >>> root@usc01a-032-16a:/home/gmckee# ip link show enp7s0f0_4 >> >>> 32: enp7s0f0_4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq >> >>> state UP mode DEFAULT group default qlen 1000 >> >>> link/ether ae:4e:85:a3:83:22 brd ff:ff:ff:ff:ff:ff >> >>> altname enp7s0f0npf0vf4 >> >>> >> >>> >> >>> the virtual function has already been assigned to a KVM VM. >> >>> >> >>> Any help is greatly appreciated . >> >> >> >> >> >> The representor plugin was developed for and tested with a SmartNIC DPU, >> >> which behaves slightly different than a system where the embedded switch >> >> is exposed to the host system. >> >> >> >> Having said that, it was developed using generic interfaces, such as >> >> devlink-port [0], so we should be able to make it work. >> >> >> >> The representor plugin looks up the representor by combining information >> >> about PF MAC (`hw_addr`) and VF number from devlink [2], a recent kernel >> >> version is required to expose the `hw_addr` attribute. >> >> >> >> A few questions: >> >> Do you see any other messages logged from the vif_plug_representor module? >> >> >> >> What kernel version is in use? >> >> >> >> Does the `hw_addr` show up for the PCI_PF flavoured port in `devlink port >> >> show`? >> >> >> >> 0: >> >> https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html >> >> 1: >> >> https://github.com/ovn-org/ovn-vif/blob/ce1a36f300a74b4eae55a7fec7d18da8b9218e29/lib/vif-plug-providers/representor/vif-plug-representor.c#L407-L469 >> >> >> >> -- >> >> Frode Nordahl >> >> >> >>> >> >>> Gav >> >>> >> >>> _______________________________________________ >> >>> discuss mailing list >> >>> disc...@openvswitch.org >> >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
