Re: tg3 driver and interrupt coalescence questions
On Tue, 2006-06-27 at 10:16 -0700, Chris A. Icide wrote: What version of the linux kernel tg3 drivers are required to support both rx and tx mitigation? ethtool -C for tg3 was added around July of 2005. The version with this change added was 3.33. What do the ethtool coalescence settings actually do (I've not been able to find actual descriptions of the different parameters in the -C section) They set the delay between the tx and rx events and the generation of interrupts for those events. These are the only parameters that are relevant for tg3: rx-frames[-irq] rx-usecs[-irq] tx-frames[-irq] tx-usecs[-irq] The frames parameters specify how many packets are received/transmitted before generating an interrupt. The usecs parameters specify how many microseconds after at least 1 packet is received/transmitted before generating an interrupt. The [-irq] parameters are the corresponding delays in updating the status when the interrupt is disabled. Is there anything special that needs to be done when compiling a kernel to enable this feature for both the kernel and the tg3 driver. No. 05:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10) Subsystem: IBM: Unknown device 02e8 Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 201 Memory at dcfe (64-bit, non-prefetchable) [size=64K] Capabilities: [40] PCI-X non-bridge device. Capabilities: [48] Power Management version 2 Capabilities: [50] Vital Product Data Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable- Linux version 2.6.9-34.ELsmp ([EMAIL PROTECTED]) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 SMP Thu Mar 9 06:23:23 GMT 2006 [EMAIL PROTECTED] ~]# ethtool -c eth1 Coalesce parameters for eth1: Adaptive RX: off TX: off stats-block-usecs: 100 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 500 rx-frames: 30 rx-usecs-irq: 500 rx-frames-irq: 20 This means that the first interrupt will be generated after 30 packets are received or 500 microseconds after the nth packet is received (1 = n 30). When irq is disabled, 20 packets instead of 30 before updating status. tx-usecs: 400 tx-frames: 53 tx-usecs-irq: 490 tx-frames-irq: 5 The first tx interrupt will be generated after 53 packets are transmitted or 400 microseconds after the nth packet is transmitted (1 = n 53). When irq is disabled, 5 packets or 490 micosecs before updating status. If the condition for generating a tx or rx interrupt is met, you get all the accumulated tx and rx status during the interrupt. Hope this helps. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: tg3 driver and interrupt coalescence questions
Chris A. Icide wrote: I've been digging around trying to get some information on the current status of interrupt mitigation features for a Braodcom 5704 interface. Specifically I'm sending and receiving lots of VoIP packets (50 pps per stream, many streams). What I can't seem to determine is this: What version of the linux kernel tg3 drivers are required to support both rx and tx mitigation? What do the ethtool coalescence settings actually do (I've not been Delay interrupts and increase individual packet latency with the intention being decreasing CPU utilization and allowing a higher aggregate packet per second limit. IE bandwidth vs latency tradeoffs. able to find actual descriptions of the different parameters in the -C section) Is there anything special that needs to be done when compiling a kernel to enable this feature for both the kernel and the tg3 driver. Are you looking to increase or decrease the settings? I would think (initially at least) that for VOIP one might not want to increase them. rick jones - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: tg3 driver and interrupt coalescence questions
Rick Jones wrote: Are you looking to increase or decrease the settings? I would think (initially at least) that for VOIP one might not want to increase them. rick jones I'm looking to decrease the interrupt load on the system. During the test I mentioned above I had some interesting and confusing results. The changes from the default settings to the settings I posted resulted in a 100% performance increase (counted by the number of VoIP audio streams the tested server could support). With default settings one of the two CPUs in the system maxed out at 99% cpu usage handling interrupts, while the second CPU was not maxed out, but we started to drop packets and the VoIP call setups started showing retransmits (which is the measurement for failure in this test) at about 300 streams. With the new settings we were able to hit 600 streams. So I definately recognized a significant improvement. However I'd still like to get more improvement. At 600 streams and 20ms packets we are looking at 30,000 pps. The % of cpu (1 CPU as apparently the interrupts can't be shared across multiple CPUs) used for interrupt handling at this 600 stream limit was 88.0%. Now what was interesting was on the test generation side (same hardware exactly) of things, I was using the SIPP software to generate the VoIP streams, and each blade in the blade server was only able to generate ~200 streams, with default settings in ethtool, one of the CPUs would hit max usage for interrupt handling at that point. So I modified the ethtool settings to match those I listed above and there was no discernable difference. It was identical performance to the default settings. Michael's response clarified for me what the actual parameters in the -C section of ethtool do, thanks Michael. However I';; be greatly appreciative of any recommedations anyone might have for interrupt mitigation settings for 100% UDP RTP traffic of 20ms packets (50 pps per stream). -Chris - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: tg3 driver and interrupt coalescence questions
Hi Cris, I'm looking to decrease the interrupt load on the system. During the test I mentioned above I had some interesting and confusing results. The changes from the default settings to the settings I posted resulted in a 100% performance increase (counted by the number of VoIP audio streams the tested server could support). With default settings one of the two CPUs in the system maxed out at 99% cpu usage handling interrupts, while the second CPU was not maxed out, but we started to drop packets and the VoIP call setups started showing retransmits (which is the measurement for failure in this test) at about 300 streams. With the new settings we were able to hit 600 streams. So I definately recognized a significant improvement. However I'd still like to get more improvement. At 600 streams and 20ms packets we are looking at 30,000 pps. The % of cpu (1 CPU as apparently the interrupts can't be shared across multiple CPUs) used for interrupt handling at this 600 stream limit was 88.0%. interrupts can be balances across multiple CPUs or not. It depends on 4 areas: 1. enabling/disabling such option in kernel upon compilation; 2. enabling/disabling of a user-space service for interrupt balancing, irqbalance on redhat, nothing such on debian; 3. enabling of disabling cpu affinity for an irq; Normally, irq-affinity for a nic interrupt is considered good, but if a CPU is overloaded you may try irq balancing. Now what was interesting was on the test generation side (same hardware exactly) of things, I was using the SIPP software to generate the VoIP streams, and each blade in the blade server was only able to generate ~200 streams, with default settings in ethtool, one of the CPUs would hit max usage for interrupt handling at that point. So I modified the ethtool settings to match those I listed above and there was no discernable difference. It was identical performance to the default settings. RTP streams generation can burn your CPU cycles as well as output of them to network, thus balancing of the load among the CPUs, irqbalancing may improve something. -- Sincerely, -- Robert Iakobashvili, coroberti at gmail dot com Navigare necesse est, vivere non est necesse. -- - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
