On Wed, 2010-01-27 at 10:49 -0500, Kelvin Ku wrote: > > > > Please post the output of: > > $ cat /proc/interrupts | grep eth > > We rename our interfaces to lan: > > $ grep lan /proc/interrupts > 61: 1 0 0 0 PCI-MSI-edge lan0 > 62: 7194004 0 0 0 PCI-MSI-edge > lan0-TxRx-0 > 63: 0 1 0 0 PCI-MSI-edge lan1 > 64: 0 0 49842410 0 PCI-MSI-edge > lan1-TxRx-0 > > Note that irqbalance is disabled. I found that it wasn't balancing IRQs like > on > our older machines. I note that the irqbalance docs say that NIC interrupts > should not be balanced, which is what we're seeing whether irqbalance is > running or not.
Interrupts look OK. (You want the interrupts for each MSI queue to land in one CPU core instead of getting bumped around. > lan1 (multicast interface) is below. Note that rx_missed_errors is non-zero. I > previously encountered this with the e1000e NIC after disabling cpuspeed, I usually disable cpuspeed, but my boxes have multiple 1 and 10GbE NICs... > which > was throttling the CPUs to 1.6 GHz (from a maximum of 2.4 GHz). I attempted to > remedy this by setting InterruptThrottleRate=0,0 in the e1000e driver, after > which we had one full day of testing with zero rx_missed_errors, but the > application still reported packet loss. rx_missed_error usually get triggered when the kernel is slow to handle incoming hardware interrupts. There's a trade-off here, increase the interrupt rate and you'll increase the kernel CPU usage as the expense of lower latency - decrease the interrupt rate, and you'll reduce the CPU usage at the expense of a higher chance of hitting the RX queue limit. I'd suggest you try setting the InterruptThrottleRate to 1000, while increasing the RX queues to 4096. (sbin/ethtool -G DEVICE rx 4096) You could try enabling multi-queue by adding IntterruptType=2, RSS=NUM_OF_QUEUE and MQ=1 to your modprobe.conf.d. > Supermicro X8DTL-iF We are using a similar SuperMicro board for 10GbE. > > Have you tried enabling pci=msi in your kernel's command line? > > No. Do I need to do this? MSI seems to be enabled: OK. looks OK. > Agreed. I ran a local netperf test and was seeing about 8 Gbps of throughput > on a single core, so this should be adequate for 1 Gbps traffic. Can you post the output of $ mpstat -P 1 ALL during peak load? - Gilboa -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
