On Mon, Jul 18, 2022 at 10:52 PM Dave Houser <davehous...@gmail.com> wrote: > > David, > > I was able to correct the DPDK NUMA warning. I am not sure what happened in > my original compiling of DPDK but the grep lines you had above were not > showing. I --reconfigured as root and was able to grep those define lines. > Now when I run ovs I don't see the NUMA warning. > I also removed the setting you suggested so the only "other_config" settings > set is `dpdk-init="true"` > I do however see the following warnings still show when starting OVS: > ``` > 2022-07-18T19:33:31.791Z|00018|dpdk|WARN|EAL: No available 1048576 kB > hugepages reported > 2022-07-18T19:33:31.791Z|00019|dpdk|WARN|EAL: No free 1048576 kB hugepages > reported on node 0 > 2022-07-18T19:33:31.791Z|00020|dpdk|WARN|EAL: No free 1048576 kB hugepages > reported on node 1 > 2022-07-18T19:33:31.791Z|00021|dpdk|WARN|EAL: No available 1048576 kB > hugepages reported > ```
Those are harmless and can be ignored. [snip] > 3) load modules and bind NIC. > This is where problems happen. I run the following commands: > > $ sudo ifconfig enp65s0f6 down > $ sudo ifconfig enp65s1f6 down > $ sudo dpdk-devbind.py -b=vfio-pci enp65s0f6 > $ sudo dpdk-devbind.py -b=vfio-pci enp65s1f6 > > The interfaces are removed from kernel control > and using dpdk-devbind.py --status the interfaces are bound to dpdk drivers First, let me detail some context. For a lot (and there is a "But", see below) of PCI devices, their respective dpdk userspace drivers need access to the pci devices resources. Such a feature is provided by a kernel module: there is vfio-pci (the recommended one), uio_pci_generic (more or less legacy) and the out of tree igb_uio that DPDK provided in the early stages of the project. To check which dpdk drivers require such system level "configuration", there is a tool in dpdk. Example, for i40e: $ dpdk-pmdinfo.py $(which ovs-vswitchd) | grep -A2 'PMD NAME: net_i40e' PMD NAME: net_i40e PMD PARAMETERS: enable_floating_veb=1floating_veb_list=<string>queue-num-per-vf=1|2|4|8|16support-multi-driver=1 PMD KMOD DEPENDENCIES: * igb_uio | uio_pci_generic | vfio-pci Now, back to your output, dpdk-devbind.py is a simple script that shows whether a pci device is bound to a _kernel_ driver. This tool does not know much about dpdk userspace drivers, themselves. > > $ sudo dpdk-devbind,py --status > Network devices using DPDK-compatible driver > ============================================ > 0000:41:00.6 'MT28850 101e' drv=vfio-pci unused=mlx5_core > 0000:41:01.6 'MT28850 101e' drv=vfio-pci unused=mlx5_core > > However when restarting ovs I encounter the following errors (this was for > only one of the interfaces, the other interface shows up in the logs with the > same errors, I just didn't include it): > ``` > 2022-07-18T20:01:46.542Z|00159|dpdk|INFO|EAL: Probe PCI driver: mlx5_pci > (15b3:101e) device: 0000:41:01.6 (socket 0) > 2022-07-18T20:01:46.544Z|00160|dpdk|WARN|mlx5_common: No Verbs device matches > PCI device 0000:41:01.6, are kernel drivers loaded? > 2022-07-18T20:01:46.544Z|00161|dpdk|ERR|mlx5_common: Verbs device not found: > 0000:41:01.6 > 2022-07-18T20:01:46.544Z|00162|dpdk|ERR|mlx5_common: Failed to initialize > device context. > 2022-07-18T20:01:46.544Z|00163|dpdk|ERR|EAL: Driver cannot attach the device > (0000:41:01.6) > 2022-07-18T20:01:46.544Z|00164|dpdk|ERR|EAL: Failed to attach device on > primary process > ``` > > Why am I getting the above errors? I can only assume the interfaces need to > be removed from the kernel and bound to vfio-pci drivers. Or should I keep > the interfaces attached to the kernel? And here it is :-). "But", mlx4/mlx5 devices are different, and must be left bound to their respective mlx kernel drivers. $ dpdk-pmdinfo.py $(which ovs-vswitchd) | grep -A1 'PMD NAME: mlx5' PMD NAME: mlx5_common_pci PMD NAME: mlx5_common_driver -- PMD NAME: mlx5_eth PMD KMOD DEPENDENCIES: * ib_uverbs & mlx5_core & mlx5_ib The mlx5 dpdk userspace driver requires the pci device to be bound to mlx5 kernel drivers. Which means that there is no special operation to do wrt kmod binding for those devices. -- David Marchand _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
