On 7/10/23 2:36 AM, josef.zahn...@swisscom.com wrote:
Ok we are getting closer to a solution. But I’m a little bit confused,
I’ll try to explain why.
I’ve removed the following sysctl values, as you stated that they are
not official FreeBSD values:
-net.inet.rss.bits="2"
-net.inet.rss.enabled="1"
-net.isr.bindthreads="1"
-net.isr.maxthreads="-1"
Without the setting above and without your proposed sysctl values, my
system was stable and the traffic distributed over all CPUs (even HT
cores). So no more drops on CARP if CPU0 has had high load. It looks
like my problem is clearly related to the “Custom” OPNsense RSS values
above. As the RSS sysctl values are noted in the OPNsense guide
(https://docs.opnsense.org/troubleshooting/performance.html
<https://docs.opnsense.org/troubleshooting/performance.html>), my
intention was, that those values needs to be set anyway for RSS. But it
seems that those 4 values are making the thing worse!
The defaults are defaults for a reason :-) and it's always a good idea
to get a baseline measurement for your workload before any tuning. It's
not possible to assess the effect of any tuning without a baseline to
compare with. Looks like everything would have worked for you even with
the OPNsense defaults although they are different from FreeBSD.
Back to your topic. You are right, queue-0 does only control traffic if
I’m enabling your sysctl values. Where can I see which protocols are
falling into control traffic (queue 0)? So your change works as
expected, thanks a lot!
There is no direct way to observe the traffic on each queue. Each rx
queue happens to have its own interrupt so I looked at the interrupt
counts (as shown by vmstat -i) and the queues' consumer index (sysctl
dev.cxl.<port>.rxq.<qid>.cidx) to test my patch.
As you stated, queue 0 is never CPU 0, which is important to know. >
I’m pretty sure that I needed the RSS values above to get RSS work on
intel NICs and with “netstat -Q” I could see the RSS queues. With
Chelsio it seems that the FreeBSD process “netisr” doesn’t do the RSS
stuff and “netstat -Q” shows nothing about RSS. For me this means that
your driver directly does the RSS magic, is this correct? Please help to
shed light on this topic.
All modern multiqueue NICs do RSS internally whether you have "options
RSS" in the kernel or not. "options RSS" enables extra code in the
kernel that gives it some control over how RSS is configured the NICs
and how many queues they should use. The net.inet.rss.* sysctls show up
only if you have "options RSS" in the kernel. Note that cxgbe does
support "options RSS" so I'm not sure why the output of "netstat -Q"
wouldn't show that. Was this the if_cxgbe.ko that you built yourself?
You must use the same configuration as the kernel to build the module.
> -net.isr.bindthreads="1"
> -net.isr.maxthreads="-1"
This doesn't show the ISR dispatch policy (net.isr.dispatch) but
whatever the policy is it applies to all the NICs in the system. If
you're comparing runs with different NICs please do it with the exact
same configuration. All NIC drivers call if_input (ether_input) to
submit work to the kernel and ether_input uses netisr to dispatch the
work. The driver doesn't make netisr calls directly.
Regards,
Navdeep
