Hi I'm developing C++ applicationthat links with DPDK 16.07 shared libraries. The application itself attaches as secondary DPDK process to primary (eg testpmd) in order to get some statistics. There were almost no DPDK related problems on Fedora 23, but when I switched to Ubuntu 16.04 I faced problems with mmap :
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ./src/snap-collector-dpdkstats EAL: Detected 28 lcore(s) PANIC in rte_eal_config_reattach(): Cannot mmap memory for rte_config 8: [./src/snap-collector-dpdkstats() [0x408389]] 7: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ffff6464830]] 6: [./src/snap-collector-dpdkstats() [0x407f5c]] 5: [./src/snap-collector-dpdkstats() [0x4088db]] 4: [./src/snap-collector-dpdkstats() [0x418e49]] 3: [/usr/lib/librte_eal.so.2.1(rte_eal_init+0xe83) [0x7ffff70b4e83]] 2: [/usr/lib/librte_eal.so.2.1(__rte_panic+0xcb) [0x7ffff70b3a25]] 1: [/usr/lib/librte_eal.so.2.1(rte_dump_stack+0x2b) [0x7ffff70bc60b]] Aborted (core dumped) After many trial and error attempts, changing linking order, excluding or including dpdk libraries, rebuilding DPDK with CONFIG_RTE_EAL_PMD_PATH variable set to pmd libraries location, I have come to conclusion: the only working workaround in my case is to run testpmd with explicitly set --base-virtaddr parameter. Please see detailed log of success and failure scenarios below. My questions are : - anyone experienced similar problems with c++ app on Ubuntu 16.04 or any other platform ? Must note that problem is reproducible only with c++ app, other pure C apps compiled and linked with gcc are working properly. Are there any other known solution for that problem ? - providing that -base-virtaddr is necessary for primary process (hovewer I doubt it) , how to determine it's value ? Currently I'm setting it to --base-virtaddr=0x2aaa4a90000 which is value taken from some other mail thread, but when I run testpmd without it, I don't see any virtual address reported by EAL output. Here's my reproduction log: ##================ setup details ================================================================== ## kernel, OS, glibc versions root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# uname -a Linux silpixa00393941 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# lsb_release -r Release: 16.04 root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ldd --version ldd (Ubuntu GLIBC 2.23-0ubuntu4) 2.23 Copyright (C) 2016 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper. ## DPDK 16.07 installation steps: * Modifications in `config/common_base`: # # Compile to share library # -CONFIG_RTE_BUILD_SHARED_LIB=n +CONFIG_RTE_BUILD_SHARED_LIB=y # Default driver path (or "" to disable) -CONFIG_RTE_EAL_PMD_PATH="" +CONFIG_RTE_EAL_PMD_PATH="/usr/lib/dpdk-pmd/" * Building target corresponding to underlying architecture, eg: make config T=x86_64-native-linuxapp-gcc make sudo make install prefix=/usr * Creation of pmd folder and symlinks for pmd drivers mkdir -p /usr/lib/dpdk-pmd find /usr/lib -type f -name 'librte_pmd*' | while read path ; do ln -s $path /usr/lib/dpdk-pmd/`echo $path | grep -o 'librte_.*so'`; done ## make log and linked libraries of the secondary process (src/snap-collector-dpdkstats) root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# make make all-recursive make[1]: Entering directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro' Making all in src make[2]: Entering directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro/src' g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT snap_collector_dpdkstats-dpdkstats.o -MD -MP -MF .deps/snap_collector_dpdkstats-dpdkstats.Tpo -c -o snap_collector_dpdkstats-dpdkstats.o `test -f 'dpdkstats.cc' || echo './'`dpdkstats.cc mv -f .deps/snap_collector_dpdkstats-dpdkstats.Tpo .deps/snap_collector_dpdkstats-dpdkstats.Po g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT snap_collector_dpdkstats-dpdkstats_collector.o -MD -MP -MF .deps/snap_collector_dpdkstats-dpdkstats_collector.Tpo -c -o snap_collector_dpdkstats-dpdkstats_collector.o `test -f 'dpdkstats_collector.cc' || echo './'`dpdkstats_collector.cc mv -f .deps/snap_collector_dpdkstats-dpdkstats_collector.Tpo .deps/snap_collector_dpdkstats-dpdkstats_collector.Po g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT utils/snap_collector_dpdkstats-dpdk.o -MD -MP -MF utils/.deps/snap_collector_dpdkstats-dpdk.Tpo -c -o utils/snap_collector_dpdkstats-dpdk.o `test -f 'utils/dpdk.cc' || echo './'`utils/dpdk.cc mv -f utils/.deps/snap_collector_dpdkstats-dpdk.Tpo utils/.deps/snap_collector_dpdkstats-dpdk.Po g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT utils/snap_collector_dpdkstats-log.o -MD -MP -MF utils/.deps/snap_collector_dpdkstats-log.Tpo -c -o utils/snap_collector_dpdkstats-log.o `test -f 'utils/log.cc' || echo './'`utils/log.cc mv -f utils/.deps/snap_collector_dpdkstats-log.Tpo utils/.deps/snap_collector_dpdkstats-log.Po g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT utils/snap_collector_dpdkstats-common.o -MD -MP -MF utils/.deps/snap_collector_dpdkstats-common.Tpo -c -o utils/snap_collector_dpdkstats-common.o `test -f 'utils/common.cc' || echo './'`utils/common.cc mv -f utils/.deps/snap_collector_dpdkstats-common.Tpo utils/.deps/snap_collector_dpdkstats-common.Po g++ -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -o snap-collector-dpdkstats snap_collector_dpdkstats-dpdkstats.o snap_collector_dpdkstats-dpdkstats_collector.o utils/snap_collector_dpdkstats-dpdk.o utils/snap_collector_dpdkstats-log.o utils/snap_collector_dpdkstats-common.o -lsnap -L/usr/local/lib -lprotobuf -pthread -lpthread -L/usr/local/lib -lgrpc++ -L/usr/lib -Wl,--as-needed -Wl,-ldpdk make[2]: Leaving directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro/src' make[2]: Entering directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro' make[2]: Leaving directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro' make[1]: Leaving directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro' root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ldd src/snap-collector-dpdkstats linux-vdso.so.1 => (0x00007ffff7ffd000) libsnap.so.0 => /usr/local/lib/libsnap.so.0 (0x00007ffff7c5e000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffff7a23000) libgrpc++.so.1 => /usr/local/lib/libgrpc++.so.1 (0x00007ffff779d000) libethdev.so.4.1 => /usr/lib/libethdev.so.4.1 (0x00007ffff750f000) librte_eal.so.2.1 => /usr/lib/librte_eal.so.2.1 (0x00007ffff72b1000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffff6f2e000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffff6c25000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffff6a0f000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffff6645000) /lib64/ld-linux-x86-64.so.2 (0x0000555555554000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffff642b000) libgrpc.so.1 => /usr/local/lib/libgrpc.so.1 (0x00007ffff617c000) librte_mempool.so.2.1 => /usr/lib/librte_mempool.so.2.1 (0x00007ffff5f74000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffff5d70000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffff5b68000) libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007ffff58fe000) libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007ffff54ba000) librte_ring.so.1.1 => /usr/lib/librte_ring.so.1.1 (0x00007ffff52b7000) ##=========SUCCESS with explicitly set --base-virtaddr =============================================== ## in testpmd terminal window root at silpixa00393941:~/kmatczax# $RTE_SDK/tools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:06:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' drv=igb_uio unused=i40e Network devices using kernel driver =================================== 0000:03:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=eth0 drv=ixgbe unused=igb_uio *Active* 0000:03:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=eth1 drv=ixgbe unused=igb_uio Other network devices ===================== <none> root at silpixa00393941:~/kmatczax# $RTE_SDK/build/app/testpmd -c 0xf -n 2 -v --base-virtaddr=0x2aaa4a90000 --socket-mem 1024,0 -- --burst=64 EAL: Detected 28 lcore(s) EAL: RTE Version: 'DPDK 16.07.0' EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size EAL: Probing VFIO support... PMD: bnxt_rte_pmd_init() called for (null) EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:1528 rte_ixgbe_pmd EAL: PCI device 0000:03:00.1 on NUMA socket 0 EAL: probe driver: 8086:1528 rte_ixgbe_pmd EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1584 rte_i40e_pmd PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001dc9 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained Configuring Port 0 (socket 0) Port 0: 68:05:CA:2D:41:C0 Checking link statuses... Port 0 Link Down Done No commandline core given, start packet forwarding Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support disabled, MP over anonymous pages disabled Logical Core 1 (socket 0) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 io packet forwarding - CRC stripping disabled - packets/burst=64 nb forwarding cores=1 - nb forwarding ports=1 RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 Press enter to exit ## switched to other terminal window for running secondary process root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ./src/snap-collector-dpdkstats EAL: Detected 28 lcore(s) EAL: Probing VFIO support... PMD: bnxt_rte_pmd_init() called for (null) EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:1528 rte_ixgbe_pmd EAL: PCI device 0000:03:00.1 on NUMA socket 0 EAL: probe driver: 8086:1528 rte_ixgbe_pmd EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1584 rte_i40e_pmd ## EAL params of secondary process Nov 16 11:17:21 silpixa00393941 dpdkstats[68951]: EAL init params: snap-collector-dpdkstats -c 0xf0 -n 2 --base-virtaddr 0x2aaa4a90000 --proc-type secondary Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: PMD: bnxt_rte_pmd_init() called for (null) Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: PCI device 0000:03:00.0 on NUMA socket 0 Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: PCI device 0000:03:00.1 on NUMA socket 0 Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: PCI device 0000:06:00.0 on NUMA socket 0 Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: probe driver: 8086:1584 rte_i40e_pmd Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: snap-collector-dpdkstats:dpdk_helper_eal_init:291 DPDK_HELPER_INITIALIZING_EAL (done) ## ========= FAILURE with no --base-virtaddr ============================================= ## in testpmd terminal window root at silpixa00393941:~/kmatczax# $RTE_SDK/build/app/testpmd -c 0xf -n 2 -v --socket-mem 1024,0 -- --burst=64 EAL: Detected 28 lcore(s) EAL: RTE Version: 'DPDK 16.07.0' EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size EAL: Probing VFIO support... PMD: bnxt_rte_pmd_init() called for (null) EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:1528 rte_ixgbe_pmd EAL: PCI device 0000:03:00.1 on NUMA socket 0 EAL: probe driver: 8086:1528 rte_ixgbe_pmd EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1584 rte_i40e_pmd PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001dc9 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained Configuring Port 0 (socket 0) Port 0: 68:05:CA:2D:41:C0 Checking link statuses... Port 0 Link Down Done No commandline core given, start packet forwarding Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support disabled, MP over anonymous pages disabled Logical Core 1 (socket 0) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 io packet forwarding - CRC stripping disabled - packets/burst=64 nb forwarding cores=1 - nb forwarding ports=1 RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 Press enter to exit ## secondary process window root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ./src/snap-collector-dpdkstats EAL: Detected 28 lcore(s) PANIC in rte_eal_config_reattach(): Cannot mmap memory for rte_config 8: [./src/snap-collector-dpdkstats() [0x408389]] 7: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ffff6464830]] 6: [./src/snap-collector-dpdkstats() [0x407f5c]] 5: [./src/snap-collector-dpdkstats() [0x4088db]] 4: [./src/snap-collector-dpdkstats() [0x418e49]] 3: [/usr/lib/librte_eal.so.2.1(rte_eal_init+0xe83) [0x7ffff70b4e83]] 2: [/usr/lib/librte_eal.so.2.1(__rte_panic+0xcb) [0x7ffff70b3a25]] 1: [/usr/lib/librte_eal.so.2.1(rte_dump_stack+0x2b) [0x7ffff70bc60b]] Aborted (core dumped) ##EAL params of secondary process Nov 16 11:21:38 silpixa00393941 dpdkstats[68968]: EAL init params: snap-collector-dpdkstats -c 0xf0 -n 2 --proc-type secondary Thanks in advance for help Krzysztof -------------------------------------------------------------------- Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN. Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.