Compiled numbers from
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-long/1200/console
1t1c
aes-gcm interfaces
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.5 1.7 2.3 1.5
IMIX 2.4 7.1 2.1 6.4
1518B 2.4 28.9 2.1 26.0
cbc-sha1 interfaces
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.5 1.7 2.3 1.5
IMIX 2.4 7.2 2.1 6.4
1518B 2.4 29.2 2.1 29.2
aes-gcm tunnels
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.4 1.6 0.4 0.2
IMIX 2.4 7.0 0.3 1.0
1518B 2.2 27.8 0.4 4.3
cbc-sha1 tunnels
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.5 1.7 0.4 0.3
IMIX 2.4 7.2 0.3 1.0
1518B 2.4 29.2 0.4 5.0
2t2c
aes-gcm interfaces
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 4.7 3.1 4.1 2.7
IMIX 4.5 13.6 3.8 11.4
1518B 3.2 39.3 3.2 39.2
cbc-sha1 interfaces
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 4.9 3.3 4.3 3.5
IMIX 4.9 14.5 4.0 12
1518B 3.5 42.8 3.5 43.2
aes-gcm tunnels
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 5.0 3.4 0.8 0.5
IMIX 4.6 13.7 0.6 1.8
1518B 3.2 39.2 0.8 9.4
cbc-sha1 tunnels
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 4.9 3.3 0.8 0.5
IMIX 4.7 14.1 0.3 1.0
1518B 3.5 42.8 0.8 9.4
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
Sent: Thursday, March 16, 2017 3:13 PM
To: 'Sergio Gonzalez Monroy' <[email protected]>; 'Rybalchenko,
Kirill' <[email protected]>; 'Nicolau, Radu'
<[email protected]>; Maciek Konstantynowicz (mkonstan) <[email protected]>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]>; '[email protected]' <[email protected]>; 'vpp-dev'
<[email protected]>
Subject: RE: IPsec Multi-Tunnel performance test suite failure
After fixing few nits with Sergio (thanks for help) looks like we are able to
measure multicore perf:
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-long/1200/console
tc13-64B-2t2c-ethip4ipsecscale1ip4-ip4base-interfaces-aes-gcm-ndrdisc
FINAL_RATE: 4668398.4375 pps (2x 2334199.21875 pps)
This is linear scale if compare with previous 2.5Mpps.
Thanks.
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
Sent: Thursday, March 16, 2017 12:02 PM
To: 'Sergio Gonzalez Monroy'
<[email protected]<mailto:[email protected]>>;
Rybalchenko, Kirill
<[email protected]<mailto:[email protected]>>; Nicolau,
Radu <[email protected]<mailto:[email protected]>>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; vpp-dev
<[email protected]<mailto:[email protected]>>; Maciek Konstantynowicz
(mkonstan) <[email protected]<mailto:[email protected]>>
Subject: RE: IPsec Multi-Tunnel performance test suite failure
I modified the script to produce this config (2 threads):
unix {
nodaemon
log /tmp/vpe.log
cli-listen localhost:5002
full-coredump
}
api-trace {
on
}
cpu {
main-core 19 corelist-workers 20,21
}
dpdk {
socket-mem 1024,1024
dev default {
num-rx-queues 1
}
dev 0000:88:00.1
dev 0000:88:00.0
enable-cryptodev dev 0000:86:01.0 dev 0000:86:01.1
uio-driver igb_uio
no-multi-seg
}
ip6 {
hash-buckets 2000000
heap-size 3G
}
I am sure that QAT is initialized with 32VFs (cat
/sys/bus/pci/devices/0000\:86\:00.0/sriov_numvfs show 32 on both machines), but:
DUT1 is showing:
vat# vat# DPDK Cryptodev support is disabled
DUT2 is showing:
vat# vat# worker crypto device id(type)
1 1(HW)
2 0(HW)
The setup for both DUTs is the same. Do you know what may be wrong?
In syslog I do see for both DUTs (lspci shows QAT initialized):
…
Mar 16 01:27:53 t3-sut1 kernel: [94934.751859] uio_pci_generic 0000:86:04.7:
enabling device (0000 -> 0002)
Mar 16 01:27:53 t3-sut1 kernel: [94934.751862] uio_pci_generic 0000:86:04.7: No
IRQ assigned to device: no support for interrupts?
…
Mar 16 01:27:58 t3-sut2 kernel: [94846.606256] uio_pci_generic 0000:86:04.7:
enabling device (0000 -> 0002)
Mar 16 01:27:58 t3-sut2 kernel: [94846.606259] uio_pci_generic 0000:86:04.7: No
IRQ assigned to device: no support for interrupts?
So no more crypto not found.
Any ideas?
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Sergio Gonzalez Monroy [mailto:[email protected]]
Sent: Wednesday, March 15, 2017 5:11 PM
To: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]<mailto:[email protected]>>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; vpp-dev
<[email protected]<mailto:[email protected]>>; Maciek Konstantynowicz
(mkonstan) <[email protected]<mailto:[email protected]>>; Rybalchenko, Kirill
<[email protected]<mailto:[email protected]>>; Nicolau,
Radu <[email protected]<mailto:[email protected]>>
Subject: Re: IPsec Multi-Tunnel performance test suite failure
I reckon something like the following should do:
enable-cryptodev dev 0000:86:01.0 dev 0000:86:01.1
Sergio
On 15/03/2017 15:36, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
wrote:
So something like:
enable-cryptodev dev 0000:86:01.0
enable-cryptodev dev 0000:86:02.0
?
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Sergio Gonzalez Monroy [mailto:[email protected]]
Sent: Wednesday, March 15, 2017 4:14 PM
To: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]><mailto:[email protected]>; Rybalchenko, Kirill
<[email protected]><mailto:[email protected]>; Nicolau,
Radu <[email protected]><mailto:[email protected]>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]><mailto:[email protected]>;
[email protected]<mailto:[email protected]>; vpp-dev
<[email protected]><mailto:[email protected]>; Maciek Konstantynowicz
(mkonstan) <[email protected]><mailto:[email protected]>
Subject: Re: IPsec Multi-Tunnel performance test suite failure
My bad.
I thought the test was already using two QAT VFs. Each workers needs one QAT VF.
Sergio
On 15/03/2017 13:47, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
wrote:
After I run CSIT with 2 physical cores and 2 worker-threads, the HW cryptodev
is not working:
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-all/1178/console
Testing is HW is there was successful and it was initialized.
Can you please take a look?
The only change I did was adding 1 more worker threads. Initialization remains
the same and Cryptodev HW was not recognized?
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
Sent: Monday, March 13, 2017 2:19 PM
To: 'Sergio Gonzalez Monroy'
<[email protected]><mailto:[email protected]>;
Maciek Konstantynowicz (mkonstan)
<[email protected]><mailto:[email protected]>; Rybalchenko, Kirill
<[email protected]><mailto:[email protected]>; Nicolau,
Radu <[email protected]><mailto:[email protected]>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]><mailto:[email protected]>;
[email protected]<mailto:[email protected]>; vpp-dev
<[email protected]><mailto:[email protected]>
Subject: RE: IPsec Multi-Tunnel performance test suite failure
+ Looping @csit-dev @vpp-dev
I will add 2 workers/threads that is not a problem.
To avoid possible exploding of number of test, we should pick only the
representative one. Apart from implementation are there any other differences
between tunnel and interface mode?
Thanks.
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Sergio Gonzalez Monroy [mailto:[email protected]]
Sent: Monday, March 13, 2017 9:58 AM
To: Maciek Konstantynowicz (mkonstan)
<[email protected]<mailto:[email protected]>>; Peter Mikus -X (pmikus -
PANTHEON TECHNOLOGIES at Cisco) <[email protected]<mailto:[email protected]>>;
Rybalchenko, Kirill
<[email protected]<mailto:[email protected]>>; Nicolau,
Radu <[email protected]<mailto:[email protected]>>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]<mailto:[email protected]>>
Subject: Re: IPsec Multi-Tunnel performance test suite failure
First, thank you to all involved!
I reckon those numbers are in the expected range.
The current test is single thread with bidirectional traffic.
I would definitely like to see tests with 2 workers/threads, one worker for
each direction. One of the reasons is that we cannot saturate QAT with a single
worker (QAT should be able to do +40Gbps of encryption).
Would it make sense to have another set of tests with 2 workers or just update
the current tests to use 2 workers?
Regarding the difference between ipsec interface and tunnels (a.k.a. SPD), the
results are expected.
Basically, it is all about the SPD (Security Policy Database) implementation.
The "tunnels" tests use the SPD, whereas the ipsec interfaces do not.
The current SPD implementation in VPP follows the guidelines of the RFC, but
it does not scale.
The ipsec interfaces do not use the SPD at all and a single entry in the fib is
all we need to "select" the traffic to encrypt.
They effectively are different graph node paths, and even though both end up
taking the same amount of cycles (at least for decryption), the interfaces
scale much better.
Sergio
On 11/03/2017 18:36, Maciek Konstantynowicz (mkonstan) wrote:
Great Peter, thanks for this final push !
Sergio, team - are these the results you expect to see?
Why such a difference interfaces vs. tunnels at 1k scale?
aes-gcm interfaces
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.5 1.7 2.3 1.5
IMIX 2.4 7.1 2.1 6.4
1518B 2.4 28.9 2.1 26.0
cbc-sha1 interfaces
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.5 1.7 2.3 1.5
IMIX 2.4 7.2 2.1 6.4
1518B 2.4 29.2 2.1 29.2
aes-gcm tunnels
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.4 1.6 0.4 0.2
IMIX 2.4 7.0 0.3 1.0
1518B 2.2 27.8 0.4 4.3
cbc-sha1 tunnels
1 tunnel 1k tunnels
Mpps Gbps Mpps Gbps
64B 2.5 1.7 0.4 0.3
IMIX 2.4 7.2 0.3 1.0
1518B 2.4 29.2 0.4 5.0
-Maciek
On 11 Mar 2017, at 06:45, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at
Cisco) <[email protected]<mailto:[email protected]>> wrote:
IPSEC is now working. PDR and NDR results are same and can be found there:
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-all/1156/console
Plots will be updated to display IPsecHW (seems like wrong xpath eval). I will
check on Monday.
So far I will run couple more iterations to see the results
@Maciek, I think it is about time to populate all TBs with QAT. Can we
coordinate?
Peter Mikus
Engineer – Software
Cisco Systems Limited
From: Nicolau, Radu [mailto:[email protected]]
Sent: Wednesday, March 08, 2017 2:07 PM
To: Gonzalez Monroy, Sergio
<[email protected]<mailto:[email protected]>>;
Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]<mailto:[email protected]>>; Rybalchenko, Kirill
<[email protected]<mailto:[email protected]>>
Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco)
<[email protected]<mailto:[email protected]>>; Maciek Konstantynowicz
(mkonstan) <[email protected]<mailto:[email protected]>>
Subject: RE: IPsec Multi-Tunnel performance test suite failure
Hi,
I submitted a small patch to only bind QAT VFs. https://gerrit.fd.io/r/#/c/5671/
The downside is that the additional check will have to be updated for new
devices.
Regards,
Radu
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
