Frode, Thanks for the great information . What are your thoughts regarding having an ability to view the port_table from an ovn-appctl command ? I think this would be helpful to troubleshoot issues when plugging / unplugging representor devices from the integration bridge ?
Gav On Thu, 22 Dec 2022 at 19:46, Frode Nordahl <frode.nord...@canonical.com> wrote: > On Wed, Dec 21, 2022 at 4:08 PM Gavin McKee <gavmcke...@googlemail.com> > wrote: > > > > Hi Frode, > > > > Thanks for your email and for taking the time to look at this. > > You're very welcome, thank you for collecting more information. > > > Kernel version > > > > root@usc01a-032-16a:/home/gmckee# uname -r > > 5.15.0-53-generic > > At that kernel version the `hw_addr` attribute is available, however > as discussed below this does not help us for the non-DPU use case. > > > Output from the devlink port does not show a hw_addr on the physical > port . > > > > root@usc01a-032-16a:/home/gmckee# devlink port > > pci/0000:07:00.0/65535: type eth netdev enp7s0f0 flavour physical port 0 > splittable false > > pci/0000:07:00.0/1: type eth netdev enp7s0f0_0 flavour pcivf controller > 0 pfnum 0 vfnum 0 external false splittable false > > function: > > hw_addr 10:70:fd:ab:cd:01 > > pci/0000:07:00.0/2: type eth netdev enp7s0f0_1 flavour pcivf controller > 0 pfnum 0 vfnum 1 external false splittable false > > function: > > hw_addr 10:70:fd:ab:cd:02 > > pci/0000:07:00.0/3: type eth netdev enp7s0f0_2 flavour pcivf controller > 0 pfnum 0 vfnum 2 external false splittable false > > function: > > hw_addr 10:70:fd:ab:cd:03 > > pci/0000:07:00.0/4: type eth netdev enp7s0f0_3 flavour pcivf controller > 0 pfnum 0 vfnum 3 external false splittable false > > function: > > hw_addr 10:70:fd:ab:cd:04 > > pci/0000:07:00.0/5: type eth netdev enp7s0f0_4 flavour pcivf controller > 0 pfnum 0 vfnum 4 external false splittable false > > function: > > hw_addr 10:70:fd:e2:44:44 > > pci/0000:07:00.0/6: type eth netdev enp7s0f0_5 flavour pcivf controller > 0 pfnum 0 vfnum 5 external false splittable false > > function: > > hw_addr 10:70:fd:e2:a3:02 > > pci/0000:07:00.1/131071: type eth netdev enp7s0f1 flavour physical port > 1 splittable false > > pci/0000:07:00.3/196608: type eth netdev enp7s0f0v1 flavour virtual > splittable false > > pci/0000:07:00.4/262144: type eth netdev enp7s0f0v2 flavour virtual > splittable false > > > > Log messages below > > > > root@usc01a-032-16a:/home/gmckee# ovn-controller > > 2022-12-20T21:42:02Z|00001|vif_plug_representor|WARN|attempt to add > function before having knowledge about PF > > [ snip ] > > The representor plugin currently assumes the presence of PCI_PF > flavoured ports and uses them to correlate to which PF each PCI_VF > port belongs. When the ovn-controller runs on the SmartNIC DPU side of > a PCI complex, these ports represent resources presented to the host > side of the PCI complex. > > When using an accelerator card that exposes the physical ports and the > embedded switch to the host system there will be no PCI_PF ports. The > devlink-port infrastructure is still useful in this topology because > it can provide a unified way of managing subfunctions [2]. > > I guess we could try to find a way to detect this mode of operation, > or introduce an option, and then use some other method to correlate > the resources. > > At the point in time where the message is logged [3], there is a lot > of information available, so we just need to find a light weight and > compatible way of doing it. > > 2: > https://legacy.netdevconf.info/0x14/pub/papers/45/0x14-paper45-talk-paper.pdf > 3: > https://github.com/ovn-org/ovn-vif/blob/ce1a36f300a74b4eae55a7fec7d18da8b9218e29/lib/vif-plug-providers/representor/vif-plug-representor.c#L321-L328 > > -- > Frode Nordahl > > > On Wed, 21 Dec 2022 at 02:50, Frode Nordahl <frode.nord...@canonical.com> > wrote: > >> > >> Hello, Gavin, > >> > >> Thank you for your interest in the vif plug infrastructure and the > representor port plugin. See replies inline below. > >> > >> tir. 20. des. 2022, 19:29 skrev Gavin McKee via discuss < > ovs-discuss@openvswitch.org>: > >>> > >>> Hi, > >>> > >>> We are hoping someone can help with the following error message. > >>> > >>> Here we add the required options to the logical switch port in OVN > North > >>> ovn-nbctl lsp-set-options c1-sw0-p1 requested-chassis=usc01a-032-16a > vif-plug-type=representor vif-plug:representor:pf-mac=10:70:fd:df:9c:3a > vif-plug:representor:vf-num=4 > >>> > >>> When I check the ovn-controller log on the hypervisor I see the > following error message: > >>> 2022-12-20T18:24:42.815Z|00108|vif_plug|INFO|Not plugging lport > c1-sw0-p1 on direction from VIF plug provider. > >>> 2022-12-20T18:24:47.816Z|00109|vif_plug_representor|INFO|No > representor port found for lport: c1-sw0-p1 pf-mac: '10:70:fd:df:9c:3a' > vf-num: '4' > >>> > >>> Here is the information for the Mellanox Connect X6 card we are using > , you can see the mac on the physical interface is defined in the entry > vif-plug:representor:pf-mac=10:70:fd:df:9c:3a > >>> ``` > >>> root@usc01a-032-16a:/home/gmckee# ip link show enp7s0f0 > >>> 14: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc mq > master ovs-system state UP mode DEFAULT group default qlen 1000 > >>> link/ether 10:70:fd:df:9c:3a brd ff:ff:ff:ff:ff:ff > >>> vf 0 link/ether 10:70:fd:ab:cd:01 brd ff:ff:ff:ff:ff:ff, spoof > checking off, link-state disable, trust off, query_rss off > >>> vf 1 link/ether 10:70:fd:ab:cd:02 brd ff:ff:ff:ff:ff:ff, spoof > checking off, link-state disable, trust off, query_rss off > >>> vf 2 link/ether 10:70:fd:ab:cd:03 brd ff:ff:ff:ff:ff:ff, spoof > checking off, link-state disable, trust off, query_rss off > >>> vf 3 link/ether 10:70:fd:ab:cd:04 brd ff:ff:ff:ff:ff:ff, spoof > checking off, link-state disable, trust off, query_rss off > >>> vf 4 link/ether 10:70:fd:e2:44:44 brd ff:ff:ff:ff:ff:ff, spoof > checking off, link-state disable, trust off, query_rss off > >>> vf 5 link/ether 10:70:fd:e2:a3:02 brd ff:ff:ff:ff:ff:ff, spoof > checking off, link-state disable, trust off, query_rss off > >>> altname enp7s0f0np0 > >>> ``` > >>> > >>> The representor information is as follows > >>> > >>> root@usc01a-032-16a:/home/gmckee# ip link show enp7s0f0_4 > >>> 32: enp7s0f0_4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > state UP mode DEFAULT group default qlen 1000 > >>> link/ether ae:4e:85:a3:83:22 brd ff:ff:ff:ff:ff:ff > >>> altname enp7s0f0npf0vf4 > >>> > >>> > >>> the virtual function has already been assigned to a KVM VM. > >>> > >>> Any help is greatly appreciated . > >> > >> > >> The representor plugin was developed for and tested with a SmartNIC > DPU, which behaves slightly different than a system where the embedded > switch is exposed to the host system. > >> > >> Having said that, it was developed using generic interfaces, such as > devlink-port [0], so we should be able to make it work. > >> > >> The representor plugin looks up the representor by combining > information about PF MAC (`hw_addr`) and VF number from devlink [2], a > recent kernel version is required to expose the `hw_addr` attribute. > >> > >> A few questions: > >> Do you see any other messages logged from the vif_plug_representor > module? > >> > >> What kernel version is in use? > >> > >> Does the `hw_addr` show up for the PCI_PF flavoured port in `devlink > port show`? > >> > >> 0: > https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html > >> 1: > https://github.com/ovn-org/ovn-vif/blob/ce1a36f300a74b4eae55a7fec7d18da8b9218e29/lib/vif-plug-providers/representor/vif-plug-representor.c#L407-L469 > >> > >> -- > >> Frode Nordahl > >> > >>> > >>> Gav > >>> > >>> _______________________________________________ > >>> discuss mailing list > >>> disc...@openvswitch.org > >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
