Hello again, patch is applied, now netisr is not eating CPU. but performance drops around 0.2Gbps according to previous kernel.
I tried also both net.isr.maxthreads=1 and net.isr.maxthreads=4 . results are same Results are: - with CCR - 1.8Gbps top: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 14 root -16 - 0B 16K CPU5 5 1:38 100.00% [crypto returns 8] 3 root -16 - 0B 16K CPU1 1 0:58 77.83% [crypto returns 0] 11 root -92 - 0B 1120K WAIT 4 0:46 41.77% [intr{irq295: t6nex0:0a0}] 11 root -92 - 0B 1120K WAIT 8 0:37 36.83% [intr{irq297: t6nex0:0a2}] 11 root -92 - 0B 1120K WAIT 12 0:19 21.05% [intr{irq307: t6nex0:1a2}] 5376 root 23 0 23M 4348K CPU10 10 0:00 3.77% iperf -B 172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf} 5376 root 23 0 23M 4348K sbwait 3 0:00 3.72% iperf -B 172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf} 5358 root 22 0 23M 4348K sbwait 12 0:00 3.72% iperf -B 172.16.70.4 -c 172.16.68.1 -P 2 -t 20{iperf} ... - with aesni - 1.25Gbps PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 14 root -16 - 0B 16K CPU7 7 1:59 99.66% [crypto returns 8] 3 root -16 - 0B 16K CPU3 3 1:10 46.09% [crypto returns 0] 11 root -92 - 0B 1120K CPU14 14 0:45 38.65% [intr{irq297: t6nex0:0a2}] 0 root -20 - 0B 2144K - 8 0:02 20.42% [kernel{crypto_12}] 0 root -20 - 0B 2144K - 5 0:02 18.54% [kernel{crypto_7}] 0 root -20 - 0B 2144K CPU4 4 0:02 18.28% [kernel{crypto_10}] 0 root -20 - 0B 2144K - 9 0:01 17.87% [kernel{crypto_5}] 0 root -20 - 0B 2144K - 11 0:02 17.66% [kernel{crypto_14}] 0 root -20 - 0B 2144K CPU15 15 0:02 16.94% [kernel{crypto_15}] 0 root -20 - 0B 2144K CPU2 2 0:00 16.91% [kernel{crypto_4}] 11 root -92 - 0B 1120K WAIT 12 0:23 16.41% [intr{irq307: t6nex0:1a2}] 0 root -20 - 0B 2144K - 6 0:02 16.28% [kernel{crypto_13}] 0 root -20 - 0B 2144K CPU14 14 0:02 14.86% [kernel{crypto_11}] 0 root -20 - 0B 2144K - 11 0:01 7.37% [kernel{crypto_9}] 5761 root 22 0 23M 4348K sbwait 13 0:00 3.96% iperf -B 172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf} 5761 root 22 0 23M 4348K sbwait 5 0:00 3.86% iperf -B 172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf} ... - with QAT - 1.46 Gbps PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 3 root -16 - 0B 16K RUN 0 1:23 93.46% [crypto returns 0] 14 root -16 - 0B 16K crypto 2 2:10 35.31% [crypto returns 8] 11 root -92 - 0B 1664K WAIT 8 0:50 21.83% [intr{irq297: t6nex0:0a2}] 11 root -92 - 0B 1664K WAIT 12 0:25 9.72% [intr{irq307: t6nex0:1a2}] 11 root -92 - 0B 1664K WAIT 8 0:00 4.12% [intr{irq368: qat1}] 6103 root 21 0 23M 4348K sbwait 12 0:00 3.88% iperf -B 172.16.70.1 -c 172.16.68.1 -P 2 -t 20{iperf} 6124 root 22 0 23M 4348K sbwait 4 0:00 3.63% iperf -B 172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf} On Fri, Apr 30, 2021 at 11:40 PM Özkan KIRIK <ozkan.ki...@gmail.com> wrote: > Thank you Mark, > > I'm going to recompile kernel with this patch and share results. > After a few days, I can install stable/13 on same hardware and then I'll > repeat same tests. > > Thanks again, > Cheers > > On Fri, Apr 30, 2021 at 11:32 PM Mark Johnston <ma...@freebsd.org> wrote: > >> On Fri, Apr 30, 2021 at 11:11:48PM +0300, Özkan KIRIK wrote: >> > Hello, >> > >> > I'm using FreeBSD stable/12 built world on 12 April 2021. >> > my setup is: >> > [freebsd host cc0] <--------> [cc1 - same freebsd, but jail] >> > >> > without IPsec, I can achieve easily to 20Gbps. (test was run with >> different >> > source IPs using multiple iperf to scale across multiple queues) >> > My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio >> > T62100-LP-CR. >> > >> > But with IPsec, throughput is limited to 2Gbps (with ccr) and only one >> > netisr thread hits %100 cpu. >> > with aesni throughput is 1,4 Gbps >> > with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x) >> > with CCR throughput is 2,0 Gbps (t6nex0) >> > But always bottleneck is netisr. >> > >> > Is there any way to workaround this netisr bottleneck ? >> > I tried to switch net.isr.dispatch to deferred and hybrid, but >> performance >> > drops a bit. >> >> I can suggest a couple of things to try. First, we configure only one >> netisr thread by default. You can create more by setting the >> net.isr.maxthreads loader tunable. I believe netipsec selects a thread >> using the SPI so adding more threads might not help much depending on >> your configuration, but testing with e.g., maxthreads = ncpu could be >> illuminating. >> >> Second, netipsec unconditionally hands rx processing off to netisr >> threads for some reason, that's why changing the dispatch policy doesn't >> help. Maybe it's to help avoid running out of kernel stack space or to >> somehow avoid packet reordering in some case that is not clear to me. I >> tried a patch (see below) which eliminates this and it helped somewhat. >> If anyone can provide an explanation for the current behaviour I'd >> appreciate it. >> >> Could you try both approaches and report back? It would also be >> interesting to know how your results compare with 13.0, if possible. >> >> commit 618ab87449d412a74bfee4932d84a6fc17afce6c >> Author: Mark Johnston <ma...@freebsd.org> >> Date: Thu Jan 7 11:29:14 2021 -0500 >> >> netipsec: Avoid deferred dispatch on the input path >> >> diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c >> index 48acba68a1fe..98d0954c4c53 100644 >> --- a/sys/netipsec/ipsec_input.c >> +++ b/sys/netipsec/ipsec_input.c >> @@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct >> secasvar *sav, int skip, >> error = ipsec_if_input(m, sav, af); >> if (error == 0) { >> NET_EPOCH_ENTER(et); >> - error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, >> m); >> + error = netisr_dispatch_src(isr_prot, >> (uintptr_t)sav->spi, m); >> NET_EPOCH_EXIT(et); >> if (error) { >> IPSEC_ISTAT(sproto, qfull); >> @@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct >> secasvar *sav, int skip, >> error = ipsec_if_input(m, sav, af); >> if (error == 0) { >> NET_EPOCH_ENTER(et); >> - error = netisr_queue_src(isr_prot, >> + error = netisr_dispatch_src(isr_prot, >> (uintptr_t)sav->spi, m); >> NET_EPOCH_EXIT(et); >> if (error) { >> > _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"