Hi Philip,
Thank you very much for the detailed instructions!
It was still not completely straight forward to do the compilation,
because of my proxy, but I finally managed to do it. However, the
installation of the packages failed due to an unsatisfied dependency (at
least is seems to be so).
I document also my compilation steps below, just in case, if someone
else will be in the same situation of using a proxy. You can jump to the
"PROBLEM" label.
On 11/12/2022 12:52 AM, filvarga wrote:
Hi Gabor,
I would suggest using ubuntu 20.04 to build your .deb packages and
then upload them to the servers.
Basically you would do this:
1) git clone https://github.com/FDio/vpp.git && cd vpp
2) git checkout v22.06
Everything worked fine up to this point.
However, the following command did not succeed, because it wanted to
download something, but I do not have a direct Internet access on the
StarBED nodes. I use a proxy, it is set for git as follows:
root@p106:~# cat .gitconfig
[http]
proxy = http://172.16.46.241:8080
3) make install-ext-dep
The output of this command was the following:
make -C build/external install-deb
make[1]: Entering directory '/root/vpp/build/external'
make[2]: Entering directory '/root/vpp/build/external'
dpkg-buildpackage: info: source package vpp-ext-deps
dpkg-buildpackage: info: source version 22.06-7
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by VPP Dev <vpp-dev@lists.fd.io>
dpkg-buildpackage: info: host architecture amd64
dpkg-source --before-build .
debian/rules clean
make[3]: Entering directory '/root/vpp/build/external/deb'
dh clean
debian/rules override_dh_clean
make[4]: Entering directory '/root/vpp/build/external/deb'
make -C .. clean
make[5]: Entering directory '/root/vpp/build/external'
make[5]: Leaving directory '/root/vpp/build/external'
make[4]: Leaving directory '/root/vpp/build/external/deb'
make[3]: Leaving directory '/root/vpp/build/external/deb'
debian/rules build
make[3]: Entering directory '/root/vpp/build/external/deb'
dh build
dh_update_autotools_config
dh_autoreconf
create-stamp debian/debhelper-build-stamp
make[3]: Leaving directory '/root/vpp/build/external/deb'
debian/rules binary
make[3]: Entering directory '/root/vpp/build/external/deb'
dh binary
dh_testroot
dh_prep
debian/rules override_dh_install
make[4]: Entering directory '/root/vpp/build/external/deb'
make -C .. install
make[5]: Entering directory '/root/vpp/build/external'
mkdir -p downloads
Downloading http://github.com/01org/intel-ipsec-mb/archive/v1.2.tar.gz
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:02:09 --:--:-- 0
curl: (28) Failed to connect to github.com port 80: Connection timed out
make[5]: *** [packages/ipsec-mb.mk:48: downloads/v1.2.tar.gz] Error 28
make[5]: Leaving directory '/root/vpp/build/external'
make[4]: *** [debian/rules:25: override_dh_install] Error 2
make[4]: Leaving directory '/root/vpp/build/external/deb'
make[3]: *** [debian/rules:17: binary] Error 2
make[3]: Leaving directory '/root/vpp/build/external/deb'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit
status 2
make[2]: *** [Makefile:74: vpp-ext-deps_22.06-7_amd64.deb] Error 2
make[2]: Leaving directory '/root/vpp/build/external'
make[1]: *** [Makefile:81: install-deb] Error 2
make[1]: Leaving directory '/root/vpp/build/external'
make: *** [Makefile:627: install-ext-deps] Error 2
root@p106:~/vpp#
As far as I understand, I should set the proxy for your downloader,
which seems to be curl. So, I tried with:
export http_proxy="http://172.16.46.241:8080";
Now, the downloader seemed to go one step further, but finally it failed:
root@p106:~/vpp# make install-ext-dep
make -C build/external install-deb
make[1]: Entering directory '/root/vpp/build/external'
make[2]: Entering directory '/root/vpp/build/external'
dpkg-buildpackage: info: source package vpp-ext-deps
dpkg-buildpackage: info: source version 22.06-7
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by VPP Dev <vpp-dev@lists.fd.io>
dpkg-buildpackage: info: host architecture amd64
dpkg-source --before-build .
debian/rules clean
make[3]: Entering directory '/root/vpp/build/external/deb'
dh clean
debian/rules override_dh_clean
make[4]: Entering directory '/root/vpp/build/external/deb'
make -C .. clean
make[5]: Entering directory '/root/vpp/build/external'
make[5]: Leaving directory '/root/vpp/build/external'
make[4]: Leaving directory '/root/vpp/build/external/deb'
make[3]: Leaving directory '/root/vpp/build/external/deb'
debian/rules build
make[3]: Entering directory '/root/vpp/build/external/deb'
dh build
dh_update_autotools_config
dh_autoreconf
create-stamp debian/debhelper-build-stamp
make[3]: Leaving directory '/root/vpp/build/external/deb'
debian/rules binary
make[3]: Entering directory '/root/vpp/build/external/deb'
dh binary
dh_testroot
dh_prep
debian/rules override_dh_install
make[4]: Entering directory '/root/vpp/build/external/deb'
make -C .. install
make[5]: Entering directory '/root/vpp/build/external'
mkdir -p downloads
Downloading http://github.com/01org/intel-ipsec-mb/archive/v1.2.tar.gz
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:02:11
--:--:-- 0
curl: (28) Failed to connect to github.com port 443: Connection timed out
make[5]: *** [packages/ipsec-mb.mk:48: downloads/v1.2.tar.gz] Error 28
make[5]: Leaving directory '/root/vpp/build/external'
make[4]: *** [debian/rules:25: override_dh_install] Error 2
make[4]: Leaving directory '/root/vpp/build/external/deb'
make[3]: *** [debian/rules:17: binary] Error 2
make[3]: Leaving directory '/root/vpp/build/external/deb'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit
status 2
make[2]: *** [Makefile:74: vpp-ext-deps_22.06-7_amd64.deb] Error 2
make[2]: Leaving directory '/root/vpp/build/external'
make[1]: *** [Makefile:81: install-deb] Error 2
make[1]: Leaving directory '/root/vpp/build/external'
make: *** [Makefile:627: install-ext-deps] Error 2
root@p106:~/vpp#
Even though I haven't seen an "https:" in the URL, I tried (as a last
chance):
export https_proxy="http://172.16.46.241:8080";
And it worked!!! :-)
4) make install-dep
5) make pkg-deb-debug
They were all OK.
Copy all of the .deb packages from the ./build-root directory to your
server and install them manually with dpkg.
PROBLEM:
Before installing the .deb files, I just checked, if I removed the
previously used non-debug version. (Yes, I did, but I willfully left
there the configuration files as you can see below.)
root@p109:~/vpp-debug/build-root# apt list vpp
Listing... Done
vpp/now 22.06-release amd64 [residual-config]
root@p109:~/vpp-debug/build-root# dpkg -i *.deb
Selecting previously unselected package libvppinfra.
(Reading database ... 67940 files and directories currently installed.)
Preparing to unpack libvppinfra_22.06-release_amd64.deb ...
Unpacking libvppinfra (22.06-release) ...
Selecting previously unselected package libvppinfra-dev.
Preparing to unpack libvppinfra-dev_22.06-release_amd64.deb ...
Unpacking libvppinfra-dev (22.06-release) ...
Selecting previously unselected package python3-vpp-api.
Preparing to unpack python3-vpp-api_22.06-release_amd64.deb ...
Unpacking python3-vpp-api (22.06-release) ...
Selecting previously unselected package vpp.
Preparing to unpack vpp_22.06-release_amd64.deb ...
start-stop-daemon: unable to stat /usr/bin/vpp (No such file or directory)
Unpacking vpp (22.06-release) ...
Selecting previously unselected package vpp-dbg.
Preparing to unpack vpp-dbg_22.06-release_amd64.deb ...
Unpacking vpp-dbg (22.06-release) ...
Selecting previously unselected package vpp-dev.
Preparing to unpack vpp-dev_22.06-release_amd64.deb ...
Unpacking vpp-dev (22.06-release) ...
Selecting previously unselected package vpp-plugin-core.
Preparing to unpack vpp-plugin-core_22.06-release_amd64.deb ...
Unpacking vpp-plugin-core (22.06-release) ...
Selecting previously unselected package vpp-plugin-devtools.
Preparing to unpack vpp-plugin-devtools_22.06-release_amd64.deb ...
Unpacking vpp-plugin-devtools (22.06-release) ...
Selecting previously unselected package vpp-plugin-dpdk.
Preparing to unpack vpp-plugin-dpdk_22.06-release_amd64.deb ...
Unpacking vpp-plugin-dpdk (22.06-release) ...
dpkg: dependency problems prevent configuration of libvppinfra:
libvppinfra depends on libc6 (>= 2.29); however:
Version of libc6:amd64 on system is 2.28-10+deb10u1.
dpkg: error processing package libvppinfra (--install):
dependency problems - leaving unconfigured
Setting up libvppinfra-dev (22.06-release) ...
dpkg: dependency problems prevent configuration of vpp:
vpp depends on libvppinfra (= 22.06-release); however:
Package libvppinfra is not configured yet.
vpp depends on libc6 (>= 2.29); however:
Version of libc6:amd64 on system is 2.28-10+deb10u1.
dpkg: error processing package vpp (--install):
dependency problems - leaving unconfigured
Setting up vpp-dbg (22.06-release) ...
Setting up vpp-dev (22.06-release) ...
dpkg: dependency problems prevent configuration of vpp-plugin-core:
vpp-plugin-core depends on vpp (= 22.06-release); however:
Package vpp is not configured yet.
vpp-plugin-core depends on libvppinfra; however:
Package libvppinfra is not configured yet.
dpkg: error processing package vpp-plugin-core (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of vpp-plugin-devtools:
vpp-plugin-devtools depends on vpp (= 22.06-release); however:
Package vpp is not configured yet.
vpp-plugin-devtools depends on libvppinfra; however:
Package libvppinfra is not configured yet.
dpkg: error processing package vpp-plugin-devtools (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of vpp-plugin-dpdk:
vpp-plugin-dpdk depends on vpp (= 22.06-release); however:
Package vpp is not configured yet.
dpkg: error processing package vpp-plugin-dpdk (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of python3-vpp-api:
python3-vpp-api depends on vpp (= 22.06-release); however:
Package vpp is not configured yet.
dpkg: error processing package python3-vpp-api (--install):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.28-10+deb10u1) ...
Errors were encountered while processing:
libvppinfra
vpp
vpp-plugin-core
vpp-plugin-devtools
vpp-plugin-dpdk
python3-vpp-api
root@p109:~/vpp-debug/build-root#
As you can see above, the previously used and removed version of VPP was
also 22.06. And that one could be installed, and it worked well on my
system (except the segmentation fault when I tried to use stateful
NAT64). Perhaps the problem could be that the libc version of the Ubuntu
server that I used for the build was too high...
Please advise me, how to continue from this point.
Best regards,
Gábor
The easiest way for me to figure out the issue would be for you to
just attach with gdb to vpp running single-worker.
1) sudo gdb --pid=$(pidof vpp)
2) Then just hit c so VPP continues to run
3) Simulate what caused the crash
4) in the gdb execute `bt`
And just send me the result of running bt.
But if it is too much you could also configure core dumps and upload
so I can check it for you.
Best regards,
Filip Varga
st 9. 11. 2022 o 8:37 Gabor LENCSE <len...@hit.bme.hu> napísal(a):
Dear Filip,
Thank you very much for your prompt reply!
I have attached the startup.conf files for the case with 2 workers
(startup.conf-mc1wc02) and for the case when I used only the main
core (startup.conf-mc0). In both cases, 4 CPU cores (0-3) were
enabled and cores 0-2 were excluded from the Linux scheduler using
the "maxcpus=4" and "isolcpus=0-2" kernel command line parameters,
respectively.
I am new to FD.io VPP. Could you please advise me if there are
pre-built debug packages available for Debian 10, and if yes,
where can I find them?
If I need to compile them myself, could you please give me a
pointer, how I can do it?
I am currently using servers in NICT StarBED, Japan. This is a
test-bed environment, and I can download packages only using a
http or ftp proxy. (Or I can download them on my Windows laptop
and upload them through a gateway.)
Thank you very much in advance!
Best regards,
Gábor
On 11/9/2022 4:04 PM, filvarga wrote:
Hi Gabor,
I will look into it and get back to you. Meanwhile could you run
the same test with a debug build and post the results ? Maybe
even core dump. Also please post your startup.conf file
Best regards,
Filip Varga
st 9. 11. 2022 o 7:50 Gabor LENCSE <len...@hit.bme.hu> napísal(a):
Dear VPP Developers,
I am a researcher and I would like to benchmark the
performance of the stateful NAT64 implementation of FD.io VPP.
Unfortunately, VPP crashed with segmentation fault.
Some details:
I used two Dell PowerEdge R430 servers as the Tester and the
DUT (Device Under Test), two 10GbE interfaces of which were
interconnected by direct cables. On the DUT, I used Debian
Linux 10.13 with 4.19.0-20-amd64 kernel and the version of
FD.io VPP was 22.06. The following packages were installed:
libvppinfra, vpp, vpp-plugin-core, vpp-plugin-dpdk.
I used the following commands to set up Stateful NAT64:
root@p109:~/DUT-settings# cat set-vpp
vppctl set interface state TenGigabitEthernet5/0/0 up
vppctl set interface state TenGigabitEthernet5/0/1 up
vppctl set interface ip address TenGigabitEthernet5/0/0
2001:2::1/64
vppctl set interface ip address TenGigabitEthernet5/0/1
198.19.0.1/24 <http://198.19.0.1/24>
vppctl ip route add 2001:2::/64 via 2001:2::1
TenGigabitEthernet5/0/0
vppctl ip route add 198.19.0.0/24 <http://198.19.0.0/24> via
198.19.0.1 TenGigabitEthernet5/0/1
vppctl set ip neighbor static TenGigabitEthernet5/0/0
2001:2::2 a0:36:9f:74:73:64
vppctl set ip neighbor static TenGigabitEthernet5/0/1
198.19.0.2 a0:36:9f:74:73:66
vppctl set interface nat64 in TenGigabitEthernet5/0/0
vppctl set interface nat64 out TenGigabitEthernet5/0/1
vppctl nat64 add prefix 64:ff9b::/96
vppctl nat64 add pool address 198.19.0.1
As for VPP, first I used two workers, but then I also tried
without workers, using only the main core. Unfortunately, VPP
crashed in both cases, but with somewhat different messages
in the syslog. (Previously I tested both setups with IPv6
packet forwarding and they worked with an excellent
performance.)
The error messages in the syslog when I used two workers:
Nov 7 16:32:02 p109 vnet[2479]: received signal SIGSEGV, PC
0x7fa86f138168, faulting address 0x4f8
Nov 7 16:32:02 p109 vnet[2479]: #0 0x00007fa8b2158137
0x7fa8b2158137
Nov 7 16:32:02 p109 vnet[2479]: #1 0x00007fa8b2086730
0x7fa8b2086730
Nov 7 16:32:02 p109 vnet[2479]: #2 0x00007fa86f138168
0x7fa86f138168
Nov 7 16:32:02 p109 vnet[2479]: #3 0x00007fa86f11d228
0x7fa86f11d228
Nov 7 16:32:02 p109 vnet[2479]: #4 0x00007fa8b20fbe62
0x7fa8b20fbe62
Nov 7 16:32:02 p109 vnet[2479]: #5 0x00007fa8b20fda4f
vlib_worker_loop + 0x5ff
Nov 7 16:32:02 p109 vnet[2479]: #6 0x00007fa8b2135e79
vlib_worker_thread_fn + 0xa9
Nov 7 16:32:02 p109 vnet[2479]: #7 0x00007fa8b2135290
vlib_worker_thread_bootstrap_fn + 0x50
Nov 7 16:32:02 p109 vnet[2479]: #8 0x00007fa8b207bfa3
start_thread + 0xf3
Nov 7 16:32:02 p109 vnet[2479]: #9 0x00007fa8b1d75eff clone
+ 0x3f
Nov 7 16:32:02 p109 systemd[1]: vpp.service: Main process
exited, code=killed, status=6/ABRT
The error messages in the syslog when I used only the main core:
Nov 7 16:48:57 p109 vnet[2606]: received signal SIGSEGV, PC
0x7fbe1d24a168, faulting address 0x1a8
Nov 7 16:48:57 p109 vnet[2606]: #0 0x00007fbe6026a137
0x7fbe6026a137
Nov 7 16:48:57 p109 vnet[2606]: #1 0x00007fbe60198730
0x7fbe60198730
Nov 7 16:48:57 p109 vnet[2606]: #2 0x00007fbe1d24a168
0x7fbe1d24a168
Nov 7 16:48:57 p109 vnet[2606]: #3 0x00007fbe1d22f228
0x7fbe1d22f228
Nov 7 16:48:57 p109 vnet[2606]: #4 0x00007fbe6020de62
0x7fbe6020de62
Nov 7 16:48:57 p109 vnet[2606]: #5 0x00007fbe602127d1
vlib_main + 0xd41
Nov 7 16:48:57 p109 vnet[2606]: #6 0x00007fbe6026906a
0x7fbe6026906a
Nov 7 16:48:57 p109 vnet[2606]: #7 0x00007fbe60169964
0x7fbe60169964
Nov 7 16:48:57 p109 systemd[1]: vpp.service: Main process
exited, code=killed, status=6/ABRT
As for the first time I started with quite high load, I
suspected that I exhausted some sort of resources, so I tried
with much lower load, but the same thing happened even when I
sent only a single packet.
I used siitperf as Tester:
https://github.com/lencsegabor/siitperf
And I followed this methodology:
https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-benchmarking-stateful
Previously my tests were successful with the following
stateful NAT64 implementations:
- Jool
- tayga+iptables
- OpenBSD PF
Could you please help me why VPP crashes, and how I could
make it work?
Thank you very much for your help in advance!
Best regards,
Gábor Lencse
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22191): https://lists.fd.io/g/vpp-dev/message/22191
Mute This Topic: https://lists.fd.io/mt/94908130/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
