Thank you Mark,
I'm going to recompile kernel with this patch and share results.
After a few days, I can install stable/13 on same hardware and then I'll
repeat same tests.
Thanks again,
Cheers
On Fri, Apr 30, 2021 at 11:32 PM Mark Johnston <ma...@freebsd.org> wrote:
> On Fri, Apr 30, 2021 at 11:11:48PM +0300, Özkan KIRIK wrote:
> > Hello,
> >
> > I'm using FreeBSD stable/12 built world on 12 April 2021.
> > my setup is:
> > [freebsd host cc0] <--------> [cc1 - same freebsd, but jail]
> >
> > without IPsec, I can achieve easily to 20Gbps. (test was run with
> different
> > source IPs using multiple iperf to scale across multiple queues)
> > My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
> > T62100-LP-CR.
> >
> > But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
> > netisr thread hits %100 cpu.
> > with aesni throughput is 1,4 Gbps
> > with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
> > with CCR throughput is 2,0 Gbps (t6nex0)
> > But always bottleneck is netisr.
> >
> > Is there any way to workaround this netisr bottleneck ?
> > I tried to switch net.isr.dispatch to deferred and hybrid, but
> performance
> > drops a bit.
>
> I can suggest a couple of things to try. First, we configure only one
> netisr thread by default. You can create more by setting the
> net.isr.maxthreads loader tunable. I believe netipsec selects a thread
> using the SPI so adding more threads might not help much depending on
> your configuration, but testing with e.g., maxthreads = ncpu could be
> illuminating.
>
> Second, netipsec unconditionally hands rx processing off to netisr
> threads for some reason, that's why changing the dispatch policy doesn't
> help. Maybe it's to help avoid running out of kernel stack space or to
> somehow avoid packet reordering in some case that is not clear to me. I
> tried a patch (see below) which eliminates this and it helped somewhat.
> If anyone can provide an explanation for the current behaviour I'd
> appreciate it.
>
> Could you try both approaches and report back? It would also be
> interesting to know how your results compare with 13.0, if possible.
>
> commit 618ab87449d412a74bfee4932d84a6fc17afce6c
> Author: Mark Johnston <ma...@freebsd.org>
> Date: Thu Jan 7 11:29:14 2021 -0500
>
> netipsec: Avoid deferred dispatch on the input path
>
> diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c
> index 48acba68a1fe..98d0954c4c53 100644
> --- a/sys/netipsec/ipsec_input.c
> +++ b/sys/netipsec/ipsec_input.c
> @@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasvar
> *sav, int skip,
> error = ipsec_if_input(m, sav, af);
> if (error == 0) {
> NET_EPOCH_ENTER(et);
> - error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, m);
> + error = netisr_dispatch_src(isr_prot, (uintptr_t)sav->spi,
> m);
> NET_EPOCH_EXIT(et);
> if (error) {
> IPSEC_ISTAT(sproto, qfull);
> @@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasvar
> *sav, int skip,
> error = ipsec_if_input(m, sav, af);
> if (error == 0) {
> NET_EPOCH_ENTER(et);
> - error = netisr_queue_src(isr_prot,
> + error = netisr_dispatch_src(isr_prot,
> (uintptr_t)sav->spi, m);
> NET_EPOCH_EXIT(et);
> if (error) {
>
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
