Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Mohammed Hawari]

Hi Ashish,

Building DPDK with mlx support in the VPP build system is experimental and not 
well tested (especially with CentOS), as I explained in my previous email. 
Don’t hesitate to contribute patches if you want to improve that support, I’ll 
happily take a look and help.
About your issue, please remove OFED as it might conflict with the rdma-core 
built by the VPP build system. Also, you can try this patch 
https://gerrit.fd.io/r/c/vpp/+/30842  it 
might fix your issue.

Best regards

Mohammed

> On 19 Jan 2021, at 11:20, ashish.sax...@hsc.com wrote:
> 
> Hi Mohammed,
> 
> I am still not able to compile latest master with mlx5 support. Can you 
> please give me some pointers on the compilation ? I have send the compilation 
> error logs in the earlier mail.
> 
> Thanks and Regards,
> Ashish
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18547): https://lists.fd.io/g/vpp-dev/message/18547
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[ashish . saxena]

Hi Mohammed,

I am still not able to compile latest master with mlx5 support. Can you please 
give me some pointers on the compilation ? I have send the compilation error 
logs in the earlier mail.

Thanks and Regards,
Ashish

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18542): https://lists.fd.io/g/vpp-dev/message/18542
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[ashish . saxena]

Hi Mohammed,

I tried to compile vpp from the latest master branch on Cento8.2 machine , but 
the compilation is failing now when I am trying to build the external 
dependencies using the following command:
make install-ext-deps

Following are the error logs:
[1056/1917] Linking target lib/librte_cmdline.so.21.0
[1057/1917] Linking target drivers/librte_common_mlx5.so.21.0
FAILED: drivers/librte_common_mlx5.so.21.0
cc  -o drivers/librte_common_mlx5.so.21.0 
'drivers/a715181@@rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o'
 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o' 
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o'
 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o' 
-Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group 
-Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl 
-lnuma lib/librte_hash.so.21.0 lib/librte_eal.so.21.0 lib/librte_kvargs.so.21.0 
lib/librte_telemetry.so.21.0 lib/librte_ring.so.21.0 lib/librte_rcu.so.21.0 
lib/librte_pci.so.21.0 drivers/librte_bus_pci.so.21.0 lib/librte_net.so.21.0 
lib/librte_mbuf.so.21.0 lib/librte_mempool.so.21.0 
-Wl,--version-script=/opt/vpp/build/external/rpm/tmp/src-dpdk/drivers/common/mlx5/version.map
 -lpthread 
-L/opt/vpp/build/external/rpm/BUILDROOT/vpp-ext-deps-21.06-0.x86_64/opt/vpp/external/x86_64/lib64
 -l:libbnxt_re-rdmav25.a -l:libcxgb4-rdmav25.a -l:libefa.a -l:libhns-rdmav25.a 
-l:libi40iw-rdmav25.a -l:libmlx4.a -l:libmlx5.a -l:libmthca-rdmav25.a 
-l:libocrdma-rdmav25.a -l:libqedr-rdmav25.a -l:libvmw_pvrdma-rdmav25.a 
-l:libhfi1verbs-rdmav25.a -l:libipathverbs-rdmav25.a -l:librxe-rdmav25.a 
-l:libsiw-rdmav25.a -l:libibverbs.a -l:librdma_util.a -l:libccan.a 
-Wl,--end-group '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' 
-Wl,-rpath-link,/opt/vpp/build/external/rpm/tmp/build-dpdk/lib 
-Wl,-rpath-link,/opt/vpp/build/external/rpm/tmp/build-dpdk/drivers
/usr/bin/ld: cannot find -l:libbnxt_re-rdmav25.a
/usr/bin/ld: cannot find -l:libcxgb4-rdmav25.a
/usr/bin/ld: cannot find -l:libefa.a
/usr/bin/ld: cannot find -l:libhns-rdmav25.a
/usr/bin/ld: cannot find -l:libi40iw-rdmav25.a
/usr/bin/ld: cannot find -l:libmthca-rdmav25.a
/usr/bin/ld: cannot find -l:libocrdma-rdmav25.a
/usr/bin/ld: cannot find -l:libqedr-rdmav25.a
/usr/bin/ld: cannot find -l:libvmw_pvrdma-rdmav25.a
/usr/bin/ld: cannot find -l:libhfi1verbs-rdmav25.a
/usr/bin/ld: cannot find -l:libipathverbs-rdmav25.a
/usr/bin/ld: cannot find -l:libsiw-rdmav25.a
/usr/bin/ld: cannot find -l:librdma_util.a
/usr/bin/ld: cannot find -l:libccan.a
collect2: error: ld returned 1 exit status
[1058/1917] Generating symbol file 
'lib/76b5a35@@rte_reorder@sha/librte_reorder.so.21.0.symbols'
[1059/1917] Linking target lib/librte_security.so.21.0
[1060/1917] Linking target drivers/librte_common_cpt.so.21.0
[1061/1917] Linking target drivers/librte_common_qat.so.21.0
[1062/1917] Generating symbol file 
'lib/76b5a35@@rte_regexdev@sha/librte_regexdev.so.21.0.symbols'
[1063/1917] Generating symbol file 
'lib/76b5a35@@rte_sched@sha/librte_sched.so.21.0.symbols'
[1064/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_i40e@sta/net_i40e_i40e_tm.c.o'
[1065/1917] Generating symbol file 
'drivers/a715181@@rte_mempool_octeontx@sha/librte_mempool_octeontx.so.21.0.symbols'
[1066/1917] Compiling C object 
'drivers/net/i40e/base/3c1f984@@i40e_base@sta/i40e_adminq.c.o'
[1067/1917] Generating symbol file 
'lib/76b5a35@@rte_ethdev@sha/librte_ethdev.so.21.0.symbols'
[1068/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_i40e@sta/net_i40e_i40e_pf.c.o'
[1069/1917] Compiling C object 
'drivers/net/i40e/1b39a87@@i40e_avx2_lib@sta/i40e_rxtx_vec_avx2.c.o'
[1070/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_fm10k@sta/net_fm10k_fm10k_rxtx.c.o'
[1071/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_fm10k@sta/net_fm10k_fm10k_ethdev.c.o'
[1072/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_fm10k@sta/net_fm10k_fm10k_rxtx_vec.c.o'
[1073/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_i40e@sta/net_i40e_i40e_ethdev_vf.c.o'
[1074/1917] Compiling C object 
'drivers/net/i40e/base/3c1f984@@i40e_base@sta/i40e_nvm.c.o'
[1075/1917] Generating rte_common_sfc_efx.sym_chk with a meson_exe.py custom 
command
[1076/1917] Compiling C object 
'drivers/net/i40e/base/3c1f984@@i40e_base@sta/i40e_common.c.o'
[1077/1917] Compiling C object 
'drivers/a715181@@tmp_rte_net_i40e@sta/

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Mohammed Hawari]

Hi Ashish,

Support for the DPDK plugin with Mellanox cards has been restored on master 
since https://gerrit.fd.io/r/c/vpp/+/30051 
. However, I can’t stress enough that you 
should get better performance (and more stability) with the rdma plugin.
Due to the DPDK build itself relying on libibverbs for mlx support, ensuring a 
consistent build of DPDK, rdma-core, and VPP is a bit challenging, and may 
break upon build system evolutions (this is what happened when we moved to 
meson-based build for DPDK). Long story short, please use the rdma plugin if 
you can, it has better support in VPP than DPDK-MLX :)

Best regards

Mohammed


> On 7 Jan 2021, at 12:54, ashish.sax...@hsc.com wrote:
> 
> Hi Mohammed,
> 
> Thanks for the reply.
> I will try with rdma plugin, But I wants to know why support for DPDK plugin 
> with Mellanox cards has been discontinued ? It was working perfectly till 
> release 20.05 .
> 
> Thanks and Regards,
> Ashish 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18483): https://lists.fd.io/g/vpp-dev/message/18483
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[ashish . saxena]

Hi Mohammed,

Thanks for the reply.
I will try with rdma plugin, But I wants to know why support for DPDK plugin 
with Mellanox cards has been discontinued ? It was working perfectly till 
release 20.05 .

Thanks and Regards,
Ashish

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18482): https://lists.fd.io/g/vpp-dev/message/18482
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Christian Hopps]

We actually had to revert the DPDK 20.08 use to continue to have this work in 
20.09.

Isn't it a bit unexpected to have a DPDK supported NIC not work though? I'd 
think we'd want this fixed, is there any idea why it's broken?

Thanks,
Chris.

> On Nov 20, 2020, at 2:47 PM, Mohammed Hawari  wrote:
> 
> Hi Ashish,
> 
> The DPDK plugin for mlx5 NICs is not supported by the current code on master. 
> We strongly encourage you to try the rdma plugin, which offers a native VPP 
> driver for mlx5 NICs without relying on DPDK. The use of this driver is 
> documented in  src/plugins/rdma/rdma_doc.md.
> 
> Best regards
> 
> Mohammed
> 
>> On 19 Nov 2020, at 13:18, ashish.sax...@hsc.com 
>>  wrote:
>> 
>> Hi vpp devs,
>> 
>> I have been trying to run vpp on latest master branch with mlx 5 enables on 
>> centos 8.2 machine .
>> I have downloaded and installed the mellanox driver from 
>> url:https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed 
>>  and 
>> installed the following dependencies:
>> #yum install perl-Term-ANSIColor
>> #yum install tcl tk python36
>> #./mlnxofedinstall --upstream-libs --dpdk --without-fw-update
>> #/etc/init.d/openibd restart
>> 
>> I cloned the latest master branch (v21.01-rc0) (git clone -b master 
>> https://gerrit.fd.io/r/vpp )
>> I enabled mlx5 from file:build/external/packages/dpdk.mk
>> DPDK_MLX5_PMD?= y
>> DPDK_MLX5_COMMON_PMD ?= y
>> 
>> Then I followed these steps to compile vpp :
>> 
>> #make wipe-release
>> #make install-dep
>> #make install-ext-deps
>> #make build-release
>> # make pkg-rpm
>> #rpm -ivh *.rpm
>> 
>> The compilation went fine , but got the following issue while installing the 
>> rpm files:
>> #rpm -ivh *.rpm
>> error: Failed dependencies:
>>  python(abi) = 2.7 is needed by 
>> vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
>>  python-setuptools is needed by 
>> vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
>> 
>> I changed line 143 on extras/rpm/vpp.spec: From "python-setuptools" to " 
>> python3-setuptools".
>> Then , I  proceeded with re-compile, and was able to install vpp 
>> successfully after that.
>> 
>> But when starting vpp, it kept crashing with error below:
>>
>> Nov  6 15:01:44 sim01 systemd[1]: Starting Vector Packet Processing 
>> Process...
>> Nov  6 15:01:44 sim01 systemd[1]: Started Vector Packet Processing Process.
>> Nov  6 15:01:44 sim01 vpp[1557233]: /usr/bin/vpp: symbol lookup error: 
>> /usr/lib/vpp_plugins/dpdk_plugin.so: undefined symbol: ibv_fork_init
>> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Main process exited, 
>> code=exited, status=127/n/a
>> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Failed with result 
>> 'exit-code'.
>> Nov  6 15:01:48 sim01 systemd[1]: Stopped Vector Packet Processing Process.
>> 
>> How can I compile vpp with mlx so that it didn't crash .
>> 
>> Thanks and Regards,
>> Ashish
>> 
>> 
>> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18127): https://lists.fd.io/g/vpp-dev/message/18127
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Mohammed Hawari]

Hi Ashish,

The DPDK plugin for mlx5 NICs is not supported by the current code on master. 
We strongly encourage you to try the rdma plugin, which offers a native VPP 
driver for mlx5 NICs without relying on DPDK. The use of this driver is 
documented in  src/plugins/rdma/rdma_doc.md.

Best regards

Mohammed

> On 19 Nov 2020, at 13:18, ashish.sax...@hsc.com wrote:
> 
> Hi vpp devs,
> 
> I have been trying to run vpp on latest master branch with mlx 5 enables on 
> centos 8.2 machine .
> I have downloaded and installed the mellanox driver from 
> url:https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed and 
> installed the following dependencies:
> #yum install perl-Term-ANSIColor
> #yum install tcl tk python36
> #./mlnxofedinstall --upstream-libs --dpdk --without-fw-update
> #/etc/init.d/openibd restart
> 
> I cloned the latest master branch (v21.01-rc0) (git clone -b master 
> https://gerrit.fd.io/r/vpp)
> I enabled mlx5 from file:build/external/packages/dpdk.mk
> DPDK_MLX5_PMD?= y
> DPDK_MLX5_COMMON_PMD ?= y
> 
> Then I followed these steps to compile vpp :
> 
> #make wipe-release
> #make install-dep
> #make install-ext-deps
> #make build-release
> # make pkg-rpm
> #rpm -ivh *.rpm
> 
> The compilation went fine , but got the following issue while installing the 
> rpm files:
> #rpm -ivh *.rpm
> error: Failed dependencies:
>  python(abi) = 2.7 is needed by vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
>  python-setuptools is needed by vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
> 
> I changed line 143 on extras/rpm/vpp.spec: From "python-setuptools" to " 
> python3-setuptools".
> Then , I  proceeded with re-compile, and was able to install vpp successfully 
> after that.
> 
> But when starting vpp, it kept crashing with error below:
>
> Nov  6 15:01:44 sim01 systemd[1]: Starting Vector Packet Processing Process...
> Nov  6 15:01:44 sim01 systemd[1]: Started Vector Packet Processing Process.
> Nov  6 15:01:44 sim01 vpp[1557233]: /usr/bin/vpp: symbol lookup error: 
> /usr/lib/vpp_plugins/dpdk_plugin.so: undefined symbol: ibv_fork_init
> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Main process exited, 
> code=exited, status=127/n/a
> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Failed with result 'exit-code'.
> Nov  6 15:01:48 sim01 systemd[1]: Stopped Vector Packet Processing Process.
> 
> How can I compile vpp with mlx so that it didn't crash .
> 
> Thanks and Regards,
> Ashish
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18113): https://lists.fd.io/g/vpp-dev/message/18113
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[ashish . saxena]

Hi vpp devs,

I have been trying to run vpp on latest master branch with mlx 5 enables on 
centos 8.2 machine .
I have downloaded and installed the mellanox driver from 
url:https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed and 
installed the following dependencies:
#yum install perl-Term-ANSIColor
#yum install tcl tk python36
#./mlnxofedinstall --upstream-libs --dpdk --without-fw-update
#/etc/init.d/openibd restart

I cloned the latest master branch (v21.01-rc0) (git clone -b master 
https://gerrit.fd.io/r/vpp)
I enabled mlx5 from file:build/external/packages/dpdk.mk
DPDK_MLX5_PMD                ?= y
DPDK_MLX5_COMMON_PMD         ?= y

Then I followed these steps to compile vpp :

#make wipe-release
#make install-dep
#make install-ext-deps
#make build-release
# make pkg-rpm
#rpm -ivh *.rpm

The compilation went fine , but got the following issue while installing the 
rpm files:
#rpm -ivh *.rpm
error: Failed dependencies:
python(abi) = 2.7 is needed by vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
python-setuptools is needed by vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64

I changed line 143 on extras/rpm/vpp.spec: From "python-setuptools" to " 
python3-setuptools".
Then , I  proceeded with re-compile, and was able to install vpp successfully 
after that.

But when starting vpp, it kept crashing with error below:

Nov  6 15:01:44 sim01 systemd[1]: Starting Vector Packet Processing Process...
Nov  6 15:01:44 sim01 systemd[1]: Started Vector Packet Processing Process.
Nov  6 15:01:44 sim01 vpp[1557233]: */usr/bin/vpp: symbol lookup error: 
/usr/lib/vpp_plugins/dpdk_plugin.so: undefined symbol: ibv_fork_init*
Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Main process exited, 
code=exited, status=127/n/a
Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Failed with result 'exit-code'.
Nov  6 15:01:48 sim01 systemd[1]: Stopped Vector Packet Processing Process.

How can I compile vpp with mlx so that it didn't crash .

Thanks and Regards,
Ashish

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18093): https://lists.fd.io/g/vpp-dev/message/18093
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-