Hi Bill,
so I followed your advise and disabled the hyperthreading in the BIOS
(No APIC settings there).
So here's what's happening. First of all, I realized how dumb I am,
since I always looked only
on the "outside" of the wan interface when watching the throughput, so
all this time it was actually 30kpps, not 15, but still much less than
wanted.
Then, after disabling the hyperthreading, I started to see more
realistic picture with CPUs, as it started to only show 2 of them while
each has an emX taskq process. So now, when I see 35% on each CPU, the
"global" CPU is also showing 35%, while before it'd show 17%, but the 2
remaining CPUs would always show 100% idle.
Also, I believe that my loader.conf settings with
hw.em.rxd="4096"
hw.em.txd="4096"
helped with all those errors on the interfaces I used to see during the
high load, so this part is good.
Now, for the bad part. I got to a total of almost 50kpps, and that was
via 70% CPU. Which probably means that at about 70kpps or so I'd hit
100%. Which actually was a lot like what you said about Xeons (you said
they maxed out around 80kpps).
Then I looked at the rates you provided and I just want to understand
something. The emX taskq is supposed to take one of the available CPUs
and probably stick with it, right? Then if on one of the interfaces you
have a very high load, then this process will take a 100% of that CPU or
core and it will hit the limit? Do I get this right? It means also that
in your situation , while you have only 14% load on the "general" CPU,
the core that handles the em1 might actually be somewhere around 55% and
the most it will take is about 70-80kpps. In that case, what is the
solution? And if I'm wrong, how helpful will it be for me to replace the
server with the one like yours or similar? Will I benefit from more than
2 CPUs/cores? Just remember, all I need is a dual port NIC, which
handles in and out - that's it.
And the last question. I saw that even though you have Intel NICs, you
still have interrupt on CPU. My RRD graphs show "0" on the interrupt. Is
this normal? I don't have polling enabled.
Thanks,
Lenny.
P.S. basically, according to my calculations, I need ~150kpps throughput.
Bill Marquette wrote:
On Wed, Mar 18, 2009 at 7:32 AM, <five2one.le...@gmail.com> wrote:
Just as an FYI and comparison proving that FreeBSD 6.2 handles this
w/out blinking. Unfortunately, I don't have any boxes running 7.x at
this time.
bwm-ng v0.6 (probing every 0.500s), press 'h' for help
input: getifaddrs type: rate
- iface Rx Tx Total
==============================================================================
em0: 3801.54 P/s 4210.02 P/s 8011.56 P/s
em1: 19377.65 P/s 18855.49 P/s 38233.14 P/s
em2: 111.75 P/s 7231.21 P/s 7342.97 P/s
em3: 1.93 P/s 1.93 P/s 3.85 P/s
lo0: 0.00 P/s 0.00 P/s 0.00 P/s
last pid: 67441; load averages: 0.47, 0.53, 0.54
up 241+10:41:12 13:33:00
48 processes: 1 running, 28 sleeping, 19 zombie
CPU states: 0.8% user, 0.0% nice, 1.3% system, 11.5% interrupt, 86.5% idle
Mem: 22M Active, 1298M Inact, 243M Wired, 112M Buf, 1696M Free
Swap: 2048M Total, 2048M Free
This is on an HP DL385G5 with two dual-core Opteron 2218 cpu's and
dual port Intel PCI-e LC fiber cards (as well as a quad port copper
card...the PPS rates above are going over the fiber card). The
operating system version is FreeBSD 6.2 - the same kernel (and some of
the same patches) that pfSense 1.2.0 runs.
# sysctl net.isr
net.isr.direct: 0
net.isr.count: 1942552898
net.isr.directed: 0
net.isr.deferred: 1942552900
net.isr.queued: 31395
net.isr.drop: 0
net.isr.swi_count: 347040539
(all 4 nics running the same settings)
# sysctl dev.em.0
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection Version - 6.2.9
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0 handle=\_SB_.PCI0.EXB0.PES5
dev.em.0.%pnpinfo: vendor=0x8086 device=0x105f subvendor=0x8086
subdevice=0x125f class=0x020000
dev.em.0.%parent: pci5
dev.em.0.debug_info: -1
dev.em.0.stats: -1
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_processing_limit: 100
net.inet.ip.intr_queue_maxlen: 5000
net.inet.ip.intr_queue_drops: 0
--Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: support-unsubscr...@pfsense.com
For additional commands, e-mail: support-h...@pfsense.com
Commercial support available - https://portal.pfsense.org
