Hi Xiaodong,
Thank you for your reply!
I'm exactly using the VPP 22.06 installed through apt from FD.io repo. The
linux-cp and linux-nl plugins also come with the VPP from the repo.
The virtual NICs on my VMs use the virtio(assigned by "model=virtio" when
installing with virt-install). The VMs are connected through libvirtd networks
(auto-create bridges). In my experiments, I can ping M2 from M1, and the
neighbor table and routing table in VPP seem to be correct.
I'm not sure which driver the VPP is using (maybe vfio-pci?). The packet
counter looked like this:
vpp# show int GE1
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
GE1 1 up 9000/0/0/0 rx packets
3719027
rx bytes
5630430079
tx packets
1107500
tx bytes
73176221
drops
76
ip4
3718961
ip6
61
tx-error
1
vpp# show int GE2
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
GE2 2 up 9000/0/0/0 rx packets
1107520
rx bytes
73177597
tx packets
3718998
tx bytes
5630427889
drops
63
ip4
1107455
ip6
62
tx-error
1162
Could you give me more information about how can I get details of the error
packets? And the main problem is that the VPP forwarding performance is much
worse than Linux kernel (2Gbps vs 26Gbps), is there any method to improve it,
or what can I do to find the reason?
Sincerely,
Wentian
________________________________
发件人: [email protected] <[email protected]> 代表 Xiaodong Xu
<[email protected]>
发送时间: 2022年10月2日 1:05
收件人: [email protected] <[email protected]>
主题: Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse than Linux
Kernel
Which vpp version are you using in your testing? As of VPP 22.06, linux-cp and
linux-nl plugins have been supported and the binary builds are available at
FD.io repository
(https://s3-docs.fd.io/vpp/22.10/gettingstarted/installing/ubuntu.html<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs3-docs.fd.io%2Fvpp%2F22.10%2Fgettingstarted%2Finstalling%2Fubuntu.html&data=05%7C01%7C%7Cbadce86b78394ff18b8408daa3cf2dc3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638002407476279512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=y5dLTkQx%2B3E%2Bg2VEpwMos6YhNZSD0mSlOdfGR6t77tY%3D&reserved=0>).
Can you install vpp from the FD.io repo and try again? (BTW, you might want to
disable the ping plugin if linux-cp is used) I would also suggest you add
static routes to rule out any issue with FRR (in which case you don't actually
need linux-cp plugin).
In the meanwhile, I wonder what uio driver you are using for your VPP machine
(igb_uio, uio_pci_generic, or vfio-pci). Assuming you are running virtio-net
driver on the guest machine, and you are connecting the M1 and R1, R1 and M2
with linux kernel bridge.
If you still run into any issue, you may want to check the neighbor table and
routing table in the VPP system first, and maybe the interface counter as well.
Regards,
Xiaodong
On Sat, Oct 1, 2022 at 3:55 AM Bu Wentian
<[email protected]<mailto:[email protected]>> wrote:
Hi everyone,
I am a beginner of VPP, and I'm trying to use VPP+FRR on KVM VMs as routers. I
have installed VPP and FRR on Ubuntu 20.04.5 VMs, and made them running in a
seperated network namespace. I use VPP Linux-cp plugin to synchronize the route
from kernel stack to VPP. The VPP and FRR seems to work, but when I use iperf3
to test the throughput, I find the performance of VPP is not good.
I created a very simple topology to test the throughput:
M1 ----- R1(with VPP) ----- M2
M1, M2 are also Ubuntu VMs(without VPP), in different subnets. I ran iperf3
server on M1 and client on M2, but only got about 2.1Gbps throughput, which is
significantly worse than using Linux kernel as a router(about 26.1Gbps).
I made another experiment on the topology:
M1 ------ R1(with VPP) ---- R2(with VPP) ------ M2
The iperf3 result is even worse (only 1.6Gbps).
I also noticed that many retransmissions happend during the iperf3 test. If I
use Linux kernel as router rather than VPP, no retransmission will happen.
Part of iperf3 output:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 166 MBytes 1.39 Gbits/sec 23 344 KBytes
[ 5] 1.00-2.00 sec 179 MBytes 1.50 Gbits/sec 49 328 KBytes
[ 5] 2.00-3.00 sec 203 MBytes 1.70 Gbits/sec 47 352 KBytes
[ 5] 3.00-4.00 sec 203 MBytes 1.70 Gbits/sec 54 339 KBytes
[ 5] 4.00-5.00 sec 211 MBytes 1.77 Gbits/sec 59 325 KBytes
Another phenomenon I found is that when I ran iperf3 directly on the R1 and R2,
I got 0 throughput at all. The output of iperf3 is like this:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 324 KBytes 2.65 Mbits/sec 4 8.74 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 324 KBytes 266 Kbits/sec 7 sender
[ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec receiver
All my VMs use 4vcpus and 8G RAM. The host machine has 16Cores(32 Threads) and
32GB RAM.
The VMs are connected by libvirtd networks.
I installed the VPP +FRR following this tutorial:
https://ipng.ch/s/articles/2021/12/23/vpp-playground.html<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fipng.ch%2Fs%2Farticles%2F2021%2F12%2F23%2Fvpp-playground.html&data=05%7C01%7C%7Cbadce86b78394ff18b8408daa3cf2dc3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638002407476279512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XHwcWCBs%2F1Fk6mjLlul6S5d390cru%2BzjoTXFclbmg5g%3D&reserved=0>
The VPP startup.conf is in the attachment.
I want to know why the VPP throughput is worse than Linux kernel, and what can
I do to improve it (I hope it better than Linux kernel forwarding). I have
searched on google for the solution but got nothing helpful. It will be
appreciated if anyone could give me a help. Please contact me if more
information or logs are needed.
Sincerely,
Wentian Bu
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21957): https://lists.fd.io/g/vpp-dev/message/21957
Mute This Topic: https://lists.fd.io/mt/94049228/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
