On Fri, 2006-03-03 at 01:35 +0000, Robert Watson wrote:
> On Thu, 2 Mar 2006, Iasen Kostov wrote:
>
> > I'm now using a MP system (dual opteron) to do NAT for about 1500 clients
> > at
> > once at speed above 200Mbit/sec full-duplex (e.g about 400Mbit/sec) and I'm
> > using PF to do the NAT. Bad thing is that the second CPU is idle. As I can
> > see from top - about 50% of the cpu is used by irq handler for the ethernet
> > adapter (irq27: bge0 bge1 - I'm using only bge0 to route via VLANs) and
> > about 30% by the network interrupt handler. I guess that the swi1:net is
> > handling the NAT (via PF) and if swi1 and irq27 are in different handlers
> > why they don't get executed on different CPUs (second CPU is 98% idle and
> > top show that both handlers run on same CPU). Aren't both handlers in
> > different kernel threads ? If they are not - is it possible to be in
> > different threads on different CPUs ?
>
> In general, yes -- I frequently look at top -S and see the ithreads running
> on
> different CPUs from each other. As you surmise, the hardware ithread is
> handling the hardware interrupts up through link layer processing, and then
> the netisr is doing the IP layer processing including NAT. On recent
> FreeBSD,
> generally if a second CPU is idle, we will generally wake up the netisr on
> another idle CPU. However, that's a property of the scheduler, and the
> details of how that happens vary a bit by FreeBSD version. You don't include
> information on which FreeBSD version you're using. It's also worth keeping
> in
> mind that if you have idle CPU time on your first CPU even with both threads
> going as fast as the hardware is driving, it's not necessarily "better" to be
> running the two tasks on different CPUs, for reasons of memory caching --
> i.e., the second CPU won't have to cache miss and read the packets in from
> memory a second time when it begins processing the mbufs previously brought
> into memory on the first CPU by the interrupt handler.
But if both threads overload 1 of the CPUs and second stays idle ?
Performance probablly would be worse because interrput handler won't be
able to handle more interrupts or swi1 won't be able to process the
packets.
>
> So a few questions:
>
> (1) What version of FreeBSD are you running?
>
6.0-STABLE #0: Thu Dec 8 21:19:54 UTC 2005 amd64
> (2) Is your network stack running MPSAFE? "sysctl debug.mpsafenet" will
> return either 0 or 1. If you're running with certain network features,
> such as the KAME IPSEC stack, you may be running with single processor
> network stack.
>
Yep. debug.mpsafenet: 1 (this is the default I think). And no nothing
there except PF's NAT. net.inet.ip.fastforwarding: 1
> (3) Are you using SCHED_4BSD (or rather, have you changed to SCHED_ULE)?
SCHED_4BSD
>
> (4) Are you running with PREEMPTION compiled into the kernel? When a thread,
> such as the netisr, is preempted by a hardware ithread, it won't
> necessarily be bounced to the other CPU immediately.
>
PREEMPTION is compiled into the kernel.
Regards
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
