Thanks for all your insightful comments. All make sense to me, and also allowed me to think on my next moves.
Thank you so much, Ilya. If I progress in this regards, I will come back to you guys. Cheers, levi On Thu, 2023-09-21 at 12:46 +0200, Ilya Maximets wrote: > On 9/21/23 11:32, Levente Csikor wrote: > > Thanks Ilya for the response. > > > > It turned out that the latest DPDK I can manually compile on the > > Bluefield-2 without errors is v20.08. > > > > After looking at the table you pointed me, my best option was > > OvS 2.14.9 with DPDK-v19.11.13. > > > > Compilations went well without error, now I have the same problem > > when > > attaching the ports to OvS. In fact, even attaching the physical > > port > > fails. > > > > The way how I initiate OvS is using my custom tailored script > > https://github.com/cslev/nvidia-bluefield-ovs-scripts/blob/main/start_ovs.sh > > From line 223. > > > > > > After adding the physical port, ovs-vsctl show provides this: > > ``` > > Bridge ovs_dpdk_br0 > > datapath_type: netdev > > Port dpdk0 > > Interface dpdk0 > > type: dpdk > > options: {dpdk-devargs="0000:03:00.0"} > > error: "Error attaching device '0000:03:00.0' to > > DPDK" > > Port ovs_dpdk_br0 > > Interface ovs_dpdk_br0 > > type: internal > > ``` > > > > vswitchd log: > > ``` > > ... > > 2023-09-21T09:29:48.626Z|00067|bridge|INFO|bridge ovs_dpdk_br0: > > added > > interface ovs_dpdk_br0 on port 65534 > > 2023-09-21T09:29:48.627Z|00068|bridge|INFO|bridge ovs_dpdk_br0: > > using > > datapath ID 0000d62195e14c4e > > 2023-09-21T09:29:48.627Z|00069|connmgr|INFO|ovs_dpdk_br0: added > > service > > controller "punix:/usr/local/var/run/openvswitch/ovs_dpdk_br0.mgmt" > > 2023-09-21T09:29:48.718Z|00070|dpdk|ERR|EAL: Driver cannot attach > > the > > device (0000:03:00.0) > > 2023-09-21T09:29:48.718Z|00071|dpdk|ERR|EAL: Failed to attach > > device on > > primary process > > 2023-09-21T09:29:48.719Z|00072|netdev_dpdk|WARN|Error attaching > > device > > '0000:03:00.0' to DPDK > > 2023-09-21T09:29:48.719Z|00073|netdev|WARN|dpdk0: could not set > > configuration (Invalid argument) > > 2023-09-21T09:29:48.719Z|00074|dpdk|ERR|Invalid port_id=32 > > 2023-09-21T09:29:57.230Z|00075|memory|INFO|27944 kB peak resident > > set > > size after 10.3 seconds > > 2023-09-21T09:29:57.230Z|00076|memory|INFO|handlers:1 ports:1 > > revalidators:1 rules:5 > > ``` > > > > > > Is there any other log files I can look into for more informational > > debug messages? > > You may enable additional DPDK debug logs by setting --log-level > in the dpdk-extra config. However, your DPDK version is older than > a hardware you're using. Meaning, the mlx driver in DPDK 19.11 > likely just doesn't recognize the hardware and will not be able to > use it. You need to figure out how to build newer versions of DPDK. > Also, representor syntax changed a few times in the past, so your > script may not work with older versions of DPDK. > > If you can't build DPDK on the board itself, cross-compiling may be > an option: > https://doc.dpdk.org/guides/platform/bluefield.html > > Also OVS 2.14 is likely a poor choice for hardware offloading. > I would not recommend anything below 2.17. 3.1 would be a better. > > > > > > > Cheers, > > levi > > > > > > On Wed, 2023-09-20 at 17:20 +0200, Ilya Maximets wrote: > > > On 9/20/23 06:39, Levente Csikor via discuss wrote: > > > > Hi All, > > > > > > > > I have a long lasting problem I have been trying to resolve for > > > > quite > > > > some time. I am playing around with an NVidia SmartNIC > > > > (Bluefield- > > > > 2), > > > > which has OvS installed by default. It works well with the > > > > kernel > > > > driver, and even TC hardware offloading is working. > > > > > > > > I want to experiment with DPDK, though. > > > > DPDK is also installed by default on the Bluefield-2. > > > > The details of the versions are as follows: > > > > > > > > OvS 2.17.7 > > > > DPDK 22.11.1.4.2 > > > > > > > > Following the "NVidia tutorials", I manage to add the physical > > > > port > > > > as > > > > a netdev device to an OVS-DPDK bridge, however, adding the > > > > virtual > > > > function fails. > > > > > > > > More details about the commands and problem are here: > > > > https://forums.developer.nvidia.com/t/error-with-configuring-ovs-dpdk-on-bluefiled-2/256030/4 > > > > > > > > > > > > Anyway, as a last resort, I thought I give a try to install OvS > > > > and > > > > DPDK from scratch following > > > > https://docs.openvswitch.org/en/latest/intro/install/dpdk/ > > > > > > > > I used the same version for OvS and DPDK; the latter was anyway > > > > the > > > > one > > > > recommended by the OvS documentation. > > > > > > > > During the `make` process, I encounter several errors I cannot > > > > really > > > > resolve > > > > ``` > > > > In file included from lib/dp-packet.h:29, > > > > from lib/bfd.c:28: > > > > lib/netdev-dpdk.h:95:12: warning: ‘struct rte_flow_tunnel’ > > > > declared > > > > inside parameter list will not be visible outside of this > > > > definition or > > > > declaration > > > > 95 | struct rte_flow_tunnel *tunnel OVS_UNUSED, > > > > | ^~~~~~~~~~~~~~~ > > > > lib/netdev-dpdk.h:106:42: warning: ‘struct rte_flow_tunnel’ > > > > declared > > > > inside parameter list will not be visible outside of this > > > > definition or > > > > declaration > > > > 106 | struct > > > > rte_flow_tunnel > > > > *tunnel OVS_UNUSED, > > > > | > > > > ^~~~~~~~~~~~~~~ > > > > lib/netdev-dpdk.h:119:12: warning: ‘struct > > > > rte_flow_restore_info’ > > > > declared inside parameter list will not be visible outside of > > > > this > > > > definition or declaration > > > > 119 | struct rte_flow_restore_info *info OVS_UNUSED, > > > > | ^~~~~~~~~~~~~~~~~~~~~ > > > > In file included from lib/bfd.c:28: > > > > lib/dp-packet.h:61:40: error: ‘RTE_MBUF_F_RX_RSS_HASH’ > > > > undeclared > > > > here > > > > (not in a function) > > > > 61 | DEF_OL_FLAG(DP_PACKET_OL_RSS_HASH, > > > > RTE_MBUF_F_RX_RSS_HASH, > > > > 0x1), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:63:41: error: ‘RTE_MBUF_F_RX_FDIR_ID’ > > > > undeclared > > > > here > > > > (not in a function) > > > > 63 | DEF_OL_FLAG(DP_PACKET_OL_FLOW_MARK, > > > > RTE_MBUF_F_RX_FDIR_ID, > > > > 0x2), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:65:47: error: ‘RTE_MBUF_F_RX_L4_CKSUM_BAD’ > > > > undeclared > > > > here (not in a function) > > > > 65 | DEF_OL_FLAG(DP_PACKET_OL_RX_L4_CKSUM_BAD, > > > > RTE_MBUF_F_RX_L4_CKSUM_BAD, 0x4), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:67:47: error: ‘RTE_MBUF_F_RX_IP_CKSUM_BAD’ > > > > undeclared > > > > here (not in a function) > > > > 67 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_BAD, > > > > RTE_MBUF_F_RX_IP_CKSUM_BAD, 0x8), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:69:48: error: ‘RTE_MBUF_F_RX_L4_CKSUM_GOOD’ > > > > undeclared > > > > here (not in a function) > > > > 69 | DEF_OL_FLAG(DP_PACKET_OL_RX_L4_CKSUM_GOOD, > > > > RTE_MBUF_F_RX_L4_CKSUM_GOOD, > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:72:48: error: ‘RTE_MBUF_F_RX_IP_CKSUM_GOOD’ > > > > undeclared > > > > here (not in a function) > > > > 72 | DEF_OL_FLAG(DP_PACKET_OL_RX_IP_CKSUM_GOOD, > > > > RTE_MBUF_F_RX_IP_CKSUM_GOOD, > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:75:42: error: ‘RTE_MBUF_F_TX_TCP_SEG’ > > > > undeclared > > > > here > > > > (not in a function); did you mean ‘RTE_MBUF_MAX_NB_SEGS’? > > > > 75 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_SEG, > > > > RTE_MBUF_F_TX_TCP_SEG, > > > > 0x40), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:77:39: error: ‘RTE_MBUF_F_TX_IPV4’ undeclared > > > > here > > > > (not > > > > in a function) > > > > 77 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV4, > > > > RTE_MBUF_F_TX_IPV4, > > > > 0x80), > > > > | > > > > ^~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:79:39: error: ‘RTE_MBUF_F_TX_IPV6’ undeclared > > > > here > > > > (not > > > > in a function) > > > > 79 | DEF_OL_FLAG(DP_PACKET_OL_TX_IPV6, > > > > RTE_MBUF_F_TX_IPV6, > > > > 0x100), > > > > | > > > > ^~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:81:44: error: ‘RTE_MBUF_F_TX_TCP_CKSUM’ > > > > undeclared > > > > here > > > > (not in a function) > > > > 81 | DEF_OL_FLAG(DP_PACKET_OL_TX_TCP_CKSUM, > > > > RTE_MBUF_F_TX_TCP_CKSUM, 0x200), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:83:44: error: ‘RTE_MBUF_F_TX_UDP_CKSUM’ > > > > undeclared > > > > here > > > > (not in a function) > > > > 83 | DEF_OL_FLAG(DP_PACKET_OL_TX_UDP_CKSUM, > > > > RTE_MBUF_F_TX_UDP_CKSUM, 0x400), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > lib/dp-packet.h:85:45: error: ‘RTE_MBUF_F_TX_SCTP_CKSUM’ > > > > undeclared > > > > here (not in a function) > > > > 85 | DEF_OL_FLAG(DP_PACKET_OL_TX_SCTP_CKSUM, > > > > RTE_MBUF_F_TX_SCTP_CKSUM, 0x800), > > > > | > > > > ^~~~~~~~~~~~~~~~~~~~~~~~ > > > > lib/dp-packet.h:52:57: note: in definition of macro > > > > ‘DEF_OL_FLAG’ > > > > 52 | #define DEF_OL_FLAG(NAME, DPDK_DEF, GENERIC_DEF) NAME = > > > > DPDK_DEF > > > > | > > > > ^~~~~~~~ > > > > make[2]: *** [Makefile:4645: lib/bfd.lo] Error 1 > > > > make[2]: Leaving directory '/opt/ovs-2.17.7' > > > > make[1]: *** [Makefile:5409: all-recursive] Error 1 > > > > make[1]: Leaving directory '/opt/ovs-2.17.7' > > > > make: *** [Makefile:3079: all] Error 2 > > > > ``` > > > > > > > > Asking google did not help at all :( > > > > Did anyone encounter such errors? > > > > > > Nvidia is using their closed-source versions of OVS and DPDK, so > > > the > > > versions you provided are not what you need. > > > > > > Upstream OVS 2.17 works only with DPDK 21.11. See the "What DPDK > > > version > > > does each Open vSwitch release work with?" section of releases > > > FAQ: > > > https://docs.openvswitch.org/en/latest/faq/releases/ > > > > > > DPDK 22.11 is supported starting with OVS 3.1. But I would guess > > > that > > > Nvidia's DPDK 22.11 is not actually an upstream DPDK 22.11. > > > > > > Best regards, Ilya Maximets. > > > > > > > > > > > btw. the Ethernet ports/hw that the Bluefield-2 has is a > > > > Mellanox/Nvidia ConnectX-6, using the mlx5_core driver. > > > > > > > > > > > > All comments are welcome :) > > > > > > > > Thanks, > > > > Levi > > >
signature.asc
Description: This is a digitally signed message part
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
