Re: [E1000-devel] 2.6.30rc7: ksoftirqd CPU saturation (x86-64 only, not x86-32) (e1000e-related?)
On 1 Jun 2009, Jesse Brandeburg spake thusly: >> 57: 0 0 0 7654 0 0 0 0 PCI-MSI-edge >>gordianet-rx-0 >> 58: 0 0 0 0 8065 0 0 0 PCI-MSI-edge >>gordianet-tx-0 >> 59: 0 0 0 0 3 0 0 0 PCI-MSI-edge >>gordianet >> 60: 0 0 0 0 0 3576 0 0 PCI-MSI-edge >>fastnet-rx-0 >> 61: 0 0 0 0 0 2555 0 0 PCI-MSI-edge >>fastnet-tx-0 >> 62: 0 0 0 0 0 0 2 0 PCI-MSI-edge >>fastnet > > where is the e1000e interrupt here? I was expecting to see eth0/eth1 Sorry, I renamed the interfaces and forgot because I've been running with them renamed for so very long that I've forgotten that they ever had other names! They're the interrupts left in above. Not exactly line saturation, is it? >> I'd not expect that level of e1000e interrupt activity to flood the >> ksoftirqds like this, and in 32-bit mode it doesn't. >> >> So, anyone know what's going on, or how I could find out? > > when you went into 64 bit mode your kernel enabled the IOMMU/DMAR, which > means that map/unmap cycles are taking many more cycles per packet, I thought that might be so, but now I'm running in 64-bit mode with a load of pretty much zero and the out-of-tree driver: and we see ksoftirqd saturation with the in-tree driver on a completely idle box (well, it's sending the odd packet out of the network interfaces because it's headless and that's the only way I can see anything at all). (actually I thought IOMMUs were supposed to *decrease* the load of things like that. Is it because pte changes have to be propagated to the IOMMU or something? It would be nice if the configure help text gave the poor user some clue whether to turn it off or on. Presumably it's sometimes useful or it wouldn't be there...) > accounting for the increased CPU utilization. you can disable at boot > with intel_iommu=off to see if it goes back to previous behavior. Not so, it goes wrong in 32-bit mode as well: my original report was incorrect, triggered by a faulty build (where 'faulty' equals 'accidentally used the in-tree e1000e rather than the out-of-tree one'). Will try hunting backwards (unisecting?) to see if the in-tree driver *ever* worked with this card. -- OpenSolaris 2009.06 is a cutting edge operating system for enterprises looking to deploy the next generation of Solaris that includes the latest innovations from Sun and the OpenSource community. Download a copy and enjoy capabilities such as Networking, Storage and Virtualization. Go to: http://p.sf.net/sfu/opensolaris-get ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
Re: [E1000-devel] 2.6.30rc7: ksoftirqd CPU saturation (x86-64 and x86-32 both) (in-tree e1000e at fault)
On 1 Jun 2009, David Miller uttered the following:
> From: Nix
> Date: Mon, 01 Jun 2009 01:16:26 +0100
>
>> I plan to try out 2.6.29 (and back to 2.6.25 or thereabouts) tomorrow
>> and see if it ever worked: if it did I'll bisect for it (rendered tricky
>> by the out-of-tree e1000e driver, but doable: it would be easier if I
>> had a clue where the e1000-devel git tree is, if anywhere, but I still
>> have no idea despite considerable searching).
>
> Why are you using the out-of-tree e1000e driver? What's wrong
> with the one in the tree? :-)
My original speculations were wrong: it goes wrong with 32-bit as well
as 64-bit kernels. The key is that it only ever goes wrong with the
in-tree driver. When I compiled the faulty kernels (32- and 64-bit), I
forgot to copy the out-of-tree driver into place. Use the out-of-tree
driver, and the CPU hogging goes away. So this is an in-tree e1000e
driver bug, definitely. The card is an 82574L.
I could bisect for it, but I have no real evidence that the in-tree
driver *ever* worked with this card. When the machine came from the
vendor it was running 2.6.25... and the out-of-tree driver.
I'll hunt backwards a bit and see if it works in 2.6.28 or something
like that. Maybe I'll be lucky.
--- out-of-tree driver with this patch needed to compile, previously
posted to e1000-devel without comment from anyone: probably the upstream
git tree, wherever that is, already has it:
diff -u e1000e-0.5.18.3/src/kcompat.h drivers/net/e1000e/kcompat.h
--- e1000e-0.5.18.3/src/kcompat.h 2009-03-05 18:43:14.0 +
+++ drivers/net/e1000e/kcompat.h2009-05-21 23:46:04.0 +0100
@@ -141,12 +141,6 @@
#endif
#endif
-#ifndef IRQ_HANDLED
-#define irqreturn_t void
-#define IRQ_HANDLED
-#define IRQ_NONE
-#endif
-
#ifndef SET_NETDEV_DEV
#define SET_NETDEV_DEV(net, pdev)
#endif
@@ -1650,4 +1644,12 @@
#define pci_request_selected_regions_exclusive(pdev, bars, name) \
pci_request_selected_regions(pdev, bars, name)
#endif /* < 2.6.29 */
+/*/
+#if ( LINUX_VERSION_CODE < KERNEL_VERSION(2,6,30) )
+#ifndef IRQ_HANDLED
+#define irqreturn_t void
+#define IRQ_HANDLED
+#define IRQ_NONE
+#endif
+#endif /* < 2.6.30 */
#endif /* _KCOMPAT_H_ */
diff -u e1000e-0.5.18.3/src/kcompat_ethtool.c
drivers/net/e1000e/kcompat_ethtool.c
--- e1000e-0.5.18.3/src/kcompat_ethtool.c 2009-03-05 18:43:14.0
+
+++ drivers/net/e1000e/kcompat_ethtool.c2009-05-20 21:28:02.0
+0100
@@ -54,6 +54,7 @@
#include
#include
#include
+#include
#include "kcompat.h"
@@ -782,7 +783,7 @@
#define ETHTOOL_OPS_COMPAT
int ethtool_ioctl(struct ifreq *ifr)
{
- struct net_device *dev = __dev_get_by_name(ifr->ifr_name);
+ struct net_device *dev = __dev_get_by_name(&init_net, ifr->ifr_name);
void *useraddr = (void *) ifr->ifr_data;
u32 ethcmd;
--
OpenSolaris 2009.06 is a cutting edge operating system for enterprises
looking to deploy the next generation of Solaris that includes the latest
innovations from Sun and the OpenSource community. Download a copy and
enjoy capabilities such as Networking, Storage and Virtualization.
Go to: http://p.sf.net/sfu/opensolaris-get
___
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
[E1000-devel] [Fwd: [PATCH] e1000e: io_error_detected callback should return PCI_ERS_RESULT_DISCONNECT on permanent failure]
I forgot to copy this to e1000-devel when I posted it to netdev. Mike Mason Original Message Subject: [PATCH] e1000e: io_error_detected callback should return PCI_ERS_RESULT_DISCONNECT on permanent failure Date: Fri, 29 May 2009 14:16:40 -0700 From: Mike Mason To: net...@vger.kernel.org PCI drivers that implement the io_error_detected callback should return PCI_ERS_RESULT_DISCONNECT if the state passed in is pci_channel_io_perm_failure. This state is not checked in many of the network drivers. This patch fixes the omission in the e1000e driver. Signed-off-by: Mike Mason --- a/drivers/net/e1000e/netdev.c 2009-05-23 14:47:00.0 -0700 +++ b/drivers/net/e1000e/netdev.c 2009-05-29 13:43:24.0 -0700 @@ -4616,6 +4616,9 @@ static pci_ers_result_t e1000_io_error_d netif_device_detach(netdev); + if (state == pci_channel_io_perm_failure) + return PCI_ERS_RESULT_DISCONNECT; + if (netif_running(netdev)) e1000e_down(adapter); pci_disable_device(pdev); -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- OpenSolaris 2009.06 is a cutting edge operating system for enterprises looking to deploy the next generation of Solaris that includes the latest innovations from Sun and the OpenSource community. Download a copy and enjoy capabilities such as Networking, Storage and Virtualization. Go to: http://p.sf.net/sfu/opensolaris-get ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
Re: [E1000-devel] Reg Transmit and Receive Queue hashing in igb e1000 driver
hhmm.. In the driver i am using, the secret key is fixed, its a constant 40 byte value. In the RSS, after the hashing, rss looks at one indirection table (using the hash value obtained from the hashing algo) which has the index of the CPUs. Whichever the index that hash result in, that particular CPU (or queue) is selected. I am not sure, but i think this indirection table gets changed as a result of which the queue assignment changes. But i am not sure its done by the driver or whether the indirection table gets changed whenever reboot happens or it changes based upon the load on the queues. Please throw some light on this. On Mon, Jun 1, 2009 at 10:53 AM, Brandeburg, Jesse < jesse.brandeb...@intel.com> wrote: > > > On Mon, 1 Jun 2009, vikram wrote: > > I found this peculiar thing regarding the hashing in RSS. According to > the > > hashing algorithm, for a particular source and destination ip address and > > port numbers, the packets should be always hashed to the same queue since > > the hashing algorithm uses only a fixed secret key in addition to the > source > > and dest ip and port numbers for hashing. But what happens is, when i > reboot > > the machine, the queue assignment changes. Just wanted to know whether > the > > hashing takes anything in consideration in addition to the above > mentioned > > parameters? > > If the driver you're using has a get_random_bytes feeding the secret key > (as some of our older drivers did) then you'll see the "random" queue > selection after each reboot. > -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
Re: [E1000-devel] Reg Transmit and Receive Queue hashing in igb e1000 driver
On Mon, 1 Jun 2009, vikram wrote: > I found this peculiar thing regarding the hashing in RSS. According to the > hashing algorithm, for a particular source and destination ip address and > port numbers, the packets should be always hashed to the same queue since > the hashing algorithm uses only a fixed secret key in addition to the source > and dest ip and port numbers for hashing. But what happens is, when i reboot > the machine, the queue assignment changes. Just wanted to know whether the > hashing takes anything in consideration in addition to the above mentioned > parameters? If the driver you're using has a get_random_bytes feeding the secret key (as some of our older drivers did) then you'll see the "random" queue selection after each reboot. -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
[E1000-devel] Reg Transmit and Receive Queue hashing in igb e1000 driver
All, I found this peculiar thing regarding the hashing in RSS. According to the hashing algorithm, for a particular source and destination ip address and port numbers, the packets should be always hashed to the same queue since the hashing algorithm uses only a fixed secret key in addition to the source and dest ip and port numbers for hashing. But what happens is, when i reboot the machine, the queue assignment changes. Just wanted to know whether the hashing takes anything in consideration in addition to the above mentioned parameters? Regards, Vikram Neelakandan -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
Re: [E1000-devel] 2.6.30rc7: ksoftirqd CPU saturation (x86-64 only, not x86-32) (e1000e-related?)
On Sun, 31 May 2009, Nix wrote: > I've just compiled a 64-bit kernel for a couple of quad-core Nehalems > (one L5520, one Core i7) for the first time. Both were using 32-bit > kernels happily before, and one (the Core i7) is happy afterwards: but > the other sees two ksoftirqd threads saturating the CPU (well, half of > it, this being a 4-core box). > So, not particularly informative. > > CPUs 3 and 5 seem to be receiving the lion's share of the local timer > interrupts and networking interrupts (this box has twin e1000es): > >CPU0 CPU1CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > 0: 33 0 0 2377 0 0 0 0 IO-APIC-edge > timer > 1: 0 0 0 2 0 0 0 0 IO-APIC-edge > i8042 > 3: 0 0 0 3 0 0 0 0 IO-APIC-edge > 4: 0 0 0372 0 0 0 0 IO-APIC-edge > serial > 8: 0 0 0 52 0 0 0 0 IO-APIC-edge > rtc0 > 12: 0 0 0 4 0 0 0 0 IO-APIC-edge > i8042 > 16: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi > uhci_hcd:usb3 > 18: 0 0 0 0 0 0 2 0 IO-APIC-fasteoi > ehci_hcd:usb1, uhci_hcd:usb8 > 19: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi > uhci_hcd:usb5, uhci_hcd:usb7 > 20: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi > acpi > 21: 0 0 0 0 70 0 0 0 IO-APIC-fasteoi > uhci_hcd:usb4, firewire_ohci > 22: 0 0 0 0 0228 0 0 IO-APIC-fasteoi > HDA Intel > 23: 0 0 0 0 67 0 0 0 IO-APIC-fasteoi > ehci_hcd:usb2, uhci_hcd:usb6 > 39: 0 0 22405 0 0 0 0 0 IO-APIC-fasteoi > arcmsr > 48: 0 0 0 0 0 0 0 0 DMAR_MSI-edge > dmar0 > 56: 0 0 0 3961 0 0 0 0 PCI-MSI-edge > ahci > 57: 0 0 0 7654 0 0 0 0 PCI-MSI-edge > gordianet-rx-0 > 58: 0 0 0 0 8065 0 0 0 PCI-MSI-edge > gordianet-tx-0 > 59: 0 0 0 0 3 0 0 0 PCI-MSI-edge > gordianet > 60: 0 0 0 0 0 3576 0 0 PCI-MSI-edge > fastnet-rx-0 > 61: 0 0 0 0 0 2555 0 0 PCI-MSI-edge > fastnet-tx-0 > 62: 0 0 0 0 0 0 2 0 PCI-MSI-edge > fastnet where is the e1000e interrupt here? I was expecting to see eth0/eth1 > NMI: 0 0 0 0 0 0 0 0 Non-maskable > interrupts > LOC: 8437 22636528 60480 13987 113384 16365 57641 Local timer > interrupts > SPU: 0 0 0 0 0 0 0 0 Spurious > interrupts > RES: 3790556780 18 2370115893 161 Rescheduling > interrupts > CAL: 83 176 144155174179173 169 Function call > interrupts > TLB:125 9 138 4282 2231022 TLB shootdowns > TRM: 0 0 0 0 0 0 0 0 Thermal event > interrupts > THR: 0 0 0 0 0 0 0 0 Threshold APIC > interrupts > ERR: 0 > MIS: 0 > > I'd not expect that level of e1000e interrupt activity to flood the > ksoftirqds like this, and in 32-bit mode it doesn't. > > So, anyone know what's going on, or how I could find out? when you went into 64 bit mode your kernel enabled the IOMMU/DMAR, which means that map/unmap cycles are taking many more cycles per packet, accounting for the increased CPU utilization. you can disable at boot with intel_iommu=off to see if it goes back to previous behavior. There is no DMAR/IOMMU in 32 bit mode, AFAIK. Hope this helps, Jesse -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
[E1000-devel] 答复: Issue "It always showed that the cable is already linked, but in fact it has been already removed."
Dears， I also tried with ethtool eth0, the issue is same as when use mii-tool. It is point that if the interface is down, we can't get its link status. I check the latest driver we uesd, and fount that it is not get the hardware status correctly when doing ioctls with command SIOCGMIIREG and request num is MII_BMSR. And I made some change of this driver, it worked as I expected. The changes it like this case MII_BMSR: e1000_read_phy_reg(hw, PHY_STATUS, &data->val_out); //data->val_out = adapter->phy_regs.bmsr; break; -邮件原件- 发件人: Brandeburg, Jesse [mailto:jesse.brandeb...@intel.com] 发送时间: 2009年5月29日 8:35 收件人: Zhang, Xu-Ming (��旭明 ITC) 抄送: e1000-de...@lists.sf.net; jesse.brandeb...@intel.com 主题: Re: [E1000-devel] Issue "It always showed that the cable is already linked, but in fact it has been already removed." On Sun, 3 May 2009, zhang.xu-m...@inventectj.com wrote: > Dear sirs， > > I am a software engineer， working at Inventec Tianjin Company. > > I found an issue with the driver for the Intel(r) 82571 PCI-E > gigabit adapters, version 0.5.18.3, release date 3/23/2009. > > I use this driver under the linux kernel version 2.6.9-34. > > I found that the driver couldn’t detected the network link status by > using the command “mii-tool” when I plug out the network cables. It > always showed that the cable is already linked, but in fact it has been > already removed. was the interface up (ifconfig eth0 up) when doing this test? Has this test ever worked? when was the last time that it worked for you? > This issue puzzles us. We can’t use it correctly as we expected. why are you trying to use mii-tool? Please use ethtool eth0 to get state in scripts, or call the ethtool ioctl instead to get link state. -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
Re: [E1000-devel] e1000 rx_ring[0] protection
Peter, you're correct, however the tx queue interface is protected by locks (qdisc lock, netdev lock) in the stack. And newer kernel versions of e1000 don't even have the tx_ring lock any more (in the driver). On the receive side the napi polling is per cpu and protects the driver from re-entrancy using the napi struct. Jesse -Original Message- From: Peter Teoh [mailto:htmldevelo...@gmail.com] Sent: Sunday, May 31, 2009 4:46 PM To: Lal Cc: e1000-devel@lists.sourceforge.net; Kernelnewbies Subject: Re: [E1000-devel] e1000 rx_ring[0] protection interesting question/observation...never noticed this...just making a guess: when sending out.multiple CPUs may simultaneously want to send out together... but when coming back.per network card.all packet is already serialized by the hardwareso since this driver is per-piece of hardwareshould not be a problem? On Fri, May 29, 2009 at 6:49 PM, Lal wrote: > In e1000_clean function in drivers/net/e1000_main.c file, tx_ring[0] > is protected by spin lock to prevent from being cleaned by multiple > cpus simultaneously, but rx_ring[0] is not. > > Why rx_ring[0] is not protected from multiple cpus ? > > Thanks > -Lal > > -- > To unsubscribe from this list: send an email with > "unsubscribe kernelnewbies" to ecar...@nl.linux.org > Please read the FAQ at http://kernelnewbies.org/FAQ > > -- Regards, Peter Teoh -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel -- Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com ___ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
