Thanks Ilya for the response. It turned out that the latest DPDK I can manually compile on the Bluefield-2 without errors is v20.08.
After looking at the table you pointed me, my best option was OvS 2.14.9 with DPDK-v19.11.13. Compilations went well without error, now I have the same problem when attaching the ports to OvS. In fact, even attaching the physical port fails. The way how I initiate OvS is using my custom tailored script https://github.com/cslev/nvidia-bluefield-ovs-scripts/blob/main/start_ovs.sh From line 223. After adding the physical port, ovs-vsctl show provides this: ``` Bridge ovs_dpdk_br0 datapath_type: netdev Port dpdk0 Interface dpdk0 type: dpdk options: {dpdk-devargs="0000:03:00.0"} error: "Error attaching device '0000:03:00.0' to DPDK" Port ovs_dpdk_br0 Interface ovs_dpdk_br0 type: internal ``` vswitchd log: ``` ... 2023-09-21T09:29:48.626Z|00067|bridge|INFO|bridge ovs_dpdk_br0: added interface ovs_dpdk_br0 on port 65534 2023-09-21T09:29:48.627Z|00068|bridge|INFO|bridge ovs_dpdk_br0: using datapath ID 0000d62195e14c4e 2023-09-21T09:29:48.627Z|00069|connmgr|INFO|ovs_dpdk_br0: added service controller "punix:/usr/local/var/run/openvswitch/ovs_dpdk_br0.mgmt" 2023-09-21T09:29:48.718Z|00070|dpdk|ERR|EAL: Driver cannot attach the device (0000:03:00.0) 2023-09-21T09:29:48.718Z|00071|dpdk|ERR|EAL: Failed to attach device on primary process 2023-09-21T09:29:48.719Z|00072|netdev_dpdk|WARN|Error attaching device '0000:03:00.0' to DPDK 2023-09-21T09:29:48.719Z|00073|netdev|WARN|dpdk0: could not set configuration (Invalid argument) 2023-09-21T09:29:48.719Z|00074|dpdk|ERR|Invalid port_id=32 2023-09-21T09:29:57.230Z|00075|memory|INFO|27944 kB peak resident set size after 10.3 seconds 2023-09-21T09:29:57.230Z|00076|memory|INFO|handlers:1 ports:1 revalidators:1 rules:5 ``` Is there any other log files I can look into for more informational debug messages? Cheers, levi On Wed, 2023-09-20 at 17:20 +0200, Ilya Maximets wrote: > On 9/20/23 06:39, Levente Csikor via discuss wrote: > > Hi All, > > > > I have a long lasting problem I have been trying to resolve for > > quite > > some time. I am playing around with an NVidia SmartNIC (Bluefield- > > 2), > > which has OvS installed by default. It works well with the kernel > > driver, and even TC hardware offloading is working. > > > > I want to experiment with DPDK, though. > > DPDK is also installed by default on the Bluefield-2. > > The details of the versions are as follows: > > > > OvS 2.17.7 > > DPDK 22.11.1.4.2 > > > > Following the "NVidia tutorials", I manage to add the physical port > > as > > a netdev device to an OVS-DPDK bridge, however, adding the virtual > > function fails. > > > > More details about the commands and problem are here: > > https://forums.developer.nvidia.com/t/error-with-configuring-ovs-dpdk-on-bluefiled-2/256030/4 > > > > > > Anyway, as a last resort, I thought I give a try to install OvS and > > DPDK from scratch following > > https://docs.openvswitch.org/en/latest/intro/install/dpdk/ > > > > I used the same version for OvS and DPDK; the latter was anyway the > > one > > recommended by the OvS documentation. > > > > During the `make` process, I encounter several errors I cannot > > really > > resolve > > ``` > > In file included from lib/dp-packet.h:29, > > from lib/bfd.c:28: > > lib/netdev-dpdk.h:95:12: warning: ‘struct rte_flow_tunnel’ declared > > inside parameter list will not be visible outside of this > > definition or > > declaration > > 95 | struct rte_flow_tunnel *tunnel OVS_UNUSED, > > | ^~~~~~~~~~~~~~~ > > lib/netdev-dpdk.h:106:42: warning: ‘struct rte_flow_tunnel’ > > declared > > inside parameter list will not be visible outside of this > > definition or > > declaration > > 106 | struct rte_flow_tunnel > > *tunnel OVS_UNUSED, > > | ^~~~~~~~~~~~~~~ > > lib/netdev-dpdk.h:119:12: warning: ‘struct rte_flow_restore_info’ > > declared inside parameter list will not be visible outside of this > > definition or declaration > > 119 | struct rte_flow_restore_info *info OVS_UNUSED, > > | ^~~~~~~~~~~~~~~~~~~~~ > > In file included from lib/bfd.c:28: > > lib/dp-packet.h:61:40: error: ‘RTE_MBUF_F_RX_RSS_HASH’ undeclared > > here > > (not in a function) > > 61 | DEF_OL_FLAG(DP_PACKET_OL_RSS_HASH, > > RTE_MBUF_F_RX_RSS_HASH, > > 0x1), > > | > > ^~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:63:41: error: ‘RTE_MBUF_F_RX_FDIR_ID’ undeclared > > here > > (not in a function) > > 63 | DEF_OL_FLAG(DP_PACKET_OL_FLOW_MARK, > > RTE_MBUF_F_RX_FDIR_ID, > > 0x2), > > | > > ^~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:65:47: error: ‘RTE_MBUF_F_RX_L4_CKSUM_BAD’ > > undeclared > > here (not in a function) > > 65 | DEF_OL_FLAG(DP_PACKET_OL_RX_L4_CKSUM_BAD, > > RTE_MBUF_F_RX_L4_CKSUM_BAD, 0x4), > > | > > ^~~~~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:67:47: error: ‘RTE_MBUF_F_RX_IP_CKSUM_BAD’ > > undeclared > > here (not in a function) > > 67 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_BAD, > > RTE_MBUF_F_RX_IP_CKSUM_BAD, 0x8), > > | > > ^~~~~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:69:48: error: ‘RTE_MBUF_F_RX_L4_CKSUM_GOOD’ > > undeclared > > here (not in a function) > > 69 | DEF_OL_FLAG(DP_PACKET_OL_RX_L4_CKSUM_GOOD, > > RTE_MBUF_F_RX_L4_CKSUM_GOOD, > > | > > ^~~~~~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:72:48: error: ‘RTE_MBUF_F_RX_IP_CKSUM_GOOD’ > > undeclared > > here (not in a function) > > 72 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_GOOD, > > RTE_MBUF_F_RX_IP_CKSUM_GOOD, > > | > > ^~~~~~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:75:42: error: ‘RTE_MBUF_F_TX_TCP_SEG’ undeclared > > here > > (not in a function); did you mean ‘RTE_MBUF_MAX_NB_SEGS’? > > 75 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_SEG, > > RTE_MBUF_F_TX_TCP_SEG, > > 0x40), > > | > > ^~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:77:39: error: ‘RTE_MBUF_F_TX_IPV4’ undeclared here > > (not > > in a function) > > 77 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV4, RTE_MBUF_F_TX_IPV4, > > 0x80), > > | ^~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:79:39: error: ‘RTE_MBUF_F_TX_IPV6’ undeclared here > > (not > > in a function) > > 79 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV6, RTE_MBUF_F_TX_IPV6, > > 0x100), > > | ^~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:81:44: error: ‘RTE_MBUF_F_TX_TCP_CKSUM’ undeclared > > here > > (not in a function) > > 81 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_CKSUM, > > RTE_MBUF_F_TX_TCP_CKSUM, 0x200), > > | > > ^~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:83:44: error: ‘RTE_MBUF_F_TX_UDP_CKSUM’ undeclared > > here > > (not in a function) > > 83 | DEF_OL_FLAG(DP_PACKET_OL_TX_UDP_CKSUM, > > RTE_MBUF_F_TX_UDP_CKSUM, 0x400), > > | > > ^~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > lib/dp-packet.h:85:45: error: ‘RTE_MBUF_F_TX_SCTP_CKSUM’ undeclared > > here (not in a function) > > 85 | DEF_OL_FLAG(DP_PACKET_OL_TX_SCTP_CKSUM, > > RTE_MBUF_F_TX_SCTP_CKSUM, 0x800), > > | > > ^~~~~~~~~~~~~~~~~~~~~~~~ > > lib/dp-packet.h:52:57: note: in definition of macro ‘DEF_OL_FLAG’ > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > DPDK_DEF > > | > > ^~~~~~~~ > > make[2]: *** [Makefile:4645: lib/bfd.lo] Error 1 > > make[2]: Leaving directory '/opt/ovs-2.17.7' > > make[1]: *** [Makefile:5409: all-recursive] Error 1 > > make[1]: Leaving directory '/opt/ovs-2.17.7' > > make: *** [Makefile:3079: all] Error 2 > > ``` > > > > Asking google did not help at all :( > > Did anyone encounter such errors? > > Nvidia is using their closed-source versions of OVS and DPDK, so the > versions you provided are not what you need. > > Upstream OVS 2.17 works only with DPDK 21.11. See the "What DPDK > version > does each Open vSwitch release work with?" section of releases FAQ: > https://docs.openvswitch.org/en/latest/faq/releases/ > > DPDK 22.11 is supported starting with OVS 3.1. But I would guess > that > Nvidia's DPDK 22.11 is not actually an upstream DPDK 22.11. > > Best regards, Ilya Maximets. > > > > > btw. the Ethernet ports/hw that the Bluefield-2 has is a > > Mellanox/Nvidia ConnectX-6, using the mlx5_core driver. > > > > > > All comments are welcome :) > > > > Thanks, > > Levi
signature.asc
Description: This is a digitally signed message part
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
