Re: [e1000]: flow control on by default - good idea really?
On Tue, 2006-17-10 at 14:02 -0700, Auke Kok wrote: For now, we should really report the FC status in e1000 at link up time. Jamal: this should help you out for now, I'll send something like this upstream later on. Thanks - this puts you at par with the tg3 at least. On Tue, 2006-17-10 at 14:46 -0700, David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 17 Oct 2006 09:05:31 -0400 It sounds to me that ethttool needs to have this semantic fix. IOW, ethttool doesnt differentiate the two items: a) advertised parameters. b) link partner negotiated parameters. and instead #a becomes #b after negotiation. methinks this needs fixing. Dave? Jeff? The way I understand it the ethernet autonegotiation mechanisms don't really give you a way to seperate these two things. Either you negotiate the link and flow control settings, or nothing. True - but I was thinking more of the state stored in the driver either by ethtool or some other part of the driver. If i remember correctly, Donald Beckers old mii tool was able to display here's what you have configured the driver for link and flow control and these are what i advertise to link peers and here's what current negotiated link and flow control parameters with link peer That distinction doesnt exist with ethtool. Or i am missing something. cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
jamal wrote: On Tue, 2006-17-10 at 14:02 -0700, Auke Kok wrote: For now, we should really report the FC status in e1000 at link up time. Jamal: this should help you out for now, I'll send something like this upstream later on. Thanks - this puts you at par with the tg3 at least. On Tue, 2006-17-10 at 14:46 -0700, David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 17 Oct 2006 09:05:31 -0400 It sounds to me that ethttool needs to have this semantic fix. IOW, ethttool doesnt differentiate the two items: a) advertised parameters. b) link partner negotiated parameters. and instead #a becomes #b after negotiation. methinks this needs fixing. Dave? Jeff? The way I understand it the ethernet autonegotiation mechanisms don't really give you a way to seperate these two things. Either you negotiate the link and flow control settings, or nothing. True - but I was thinking more of the state stored in the driver either by ethtool or some other part of the driver. If i remember correctly, Donald Beckers old mii tool was able to display here's what you have configured the driver for link and flow control and these are what i advertise to link peers and here's what current negotiated link and flow control parameters with link peer That distinction doesnt exist with ethtool. Or i am missing something. nope, there's not even an ethtool cmd to query for that data AFAICS Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote: jamal wrote: I think when the e1000 says via ethtool rx is on - it means that it is _advertising_ flow control as opposed to detecting partner has flow control capability. Auke, can you also check this as well? Just found this in my todo box - a bit late :( yes, that appears to be the correct interpretation: we never read back the detected FC state from the hardware. It sounds to me that ethttool needs to have this semantic fix. IOW, ethttool doesnt differentiate the two items: a) advertised parameters. b) link partner negotiated parameters. and instead #a becomes #b after negotiation. methinks this needs fixing. Dave? Jeff? cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
jamal wrote: On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote: jamal wrote: I think when the e1000 says via ethtool rx is on - it means that it is _advertising_ flow control as opposed to detecting partner has flow control capability. Auke, can you also check this as well? Just found this in my todo box - a bit late :( yes, that appears to be the correct interpretation: we never read back the detected FC state from the hardware. It sounds to me that ethttool needs to have this semantic fix. IOW, ethttool doesnt differentiate the two items: a) advertised parameters. b) link partner negotiated parameters. and instead #a becomes #b after negotiation. that's odd and confusing, we should keep them separate. methinks this needs fixing. Dave? Jeff? Alternatively, we can report in the driver at link up time what FC settings were succesfull using printk. The output of `eththool eth0` would be the best place to put this output as well since that displays status and not settings which are displayed with '-a|-A' options of ethtool. Lets keep `-a|-A` the same and work on showing negotiated FC parameters in `ethtool ethX` instead. Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Tue, 17 Oct 2006 09:05:31 -0400 jamal [EMAIL PROTECTED] wrote: On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote: jamal wrote: I think when the e1000 says via ethtool rx is on - it means that it is _advertising_ flow control as opposed to detecting partner has flow control capability. Auke, can you also check this as well? Just found this in my todo box - a bit late :( yes, that appears to be the correct interpretation: we never read back the detected FC state from the hardware. It sounds to me that ethttool needs to have this semantic fix. IOW, ethttool doesnt differentiate the two items: a) advertised parameters. b) link partner negotiated parameters. and instead #a becomes #b after negotiation. methinks this needs fixing. Dave? Jeff? cheers, jamal If #a becomes #b after negotiation, that is a bug. Otherwise, if imagine that connection was moved from gigabit with flow control to 10mbit with no flow control, then back to gigabit with flow control. You would want the advertised parameters to stay the same and not change. I think driver need to keep track of advertising and status bits in separate fields. Another pet bug, is that drivers need to accept and remember ethtool settings when offline before device is brought up. The existing if_mii interface doesn't do that, and so drivers that use it are broken. -- Stephen Hemminger [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
jamal wrote:
On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote:
jamal wrote:
I think when the e1000 says via ethtool rx is on - it means that it
is _advertising_ flow control as opposed to detecting partner has flow
control capability.
Auke, can you also check this as well?
Just found this in my todo box - a bit late :(
yes, that appears to be the correct interpretation: we never read back the
detected FC state from the hardware.
For now, we should really report the FC status in e1000 at link up time. Jamal: this
should help you out for now, I'll send something like this upstream later on.
Cheers,
Auke
e1000: Display Flow Control setting used after link negotiation
No part of e1000 was reporting which Fc settings were effectively
negotiated with the link partner so that it would be impossible
to know if FC was actually used at all.
Signed-off-by: Auke Kok [EMAIL PROTECTED]
diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index ce0d35f..ff7f396 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -2426,15 +2426,22 @@ e1000_watchdog(unsigned long data)
if (link) {
if (!netif_carrier_ok(netdev)) {
+ uint32_t ctrl;
boolean_t txb2b = 1;
e1000_get_speed_and_duplex(adapter-hw,
adapter-link_speed,
adapter-link_duplex);
- DPRINTK(LINK, INFO, NIC Link is Up %d Mbps %s\n,
+ ctrl = E1000_READ_REG(adapter-hw, CTRL);
+ DPRINTK(LINK, INFO, NIC Link is Up %d Mbps %s,
+ Flow Control: %s\n,
adapter-link_speed,
adapter-link_duplex == FULL_DUPLEX ?
- Full Duplex : Half Duplex);
+ Full Duplex : Half Duplex,
+ ((ctrl E1000_CTRL_TFCE) (ctrl
+ E1000_CTRL_RFCE)) ? RX/TX : ((ctrl
+ E1000_CTRL_RFCE) ? RX : ((ctrl
+ E1000_CTRL_TFCE) ? TX : None )));
/* tweak tx_queue_len according to speed/duplex
* and adjust the timeout factor */
Re: [e1000]: flow control on by default - good idea really?
From: jamal [EMAIL PROTECTED] Date: Tue, 17 Oct 2006 09:05:31 -0400 It sounds to me that ethttool needs to have this semantic fix. IOW, ethttool doesnt differentiate the two items: a) advertised parameters. b) link partner negotiated parameters. and instead #a becomes #b after negotiation. methinks this needs fixing. Dave? Jeff? The way I understand it the ethernet autonegotiation mechanisms don't really give you a way to seperate these two things. Either you negotiate the link and flow control settings, or nothing. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Bug in e1000 + semantics of flow control WAS(Re: [e1000]: flow control on by default - good idea really?
To the good folks at Intel: Do you need more info on how to reproduce the issue below? Note, there are 2 issues - one being a larger ethtool semantic issue and the other being what i perceive to be a bug in the e1000. cheers, jamal On Thu, 2006-20-07 at 16:15 -0400, jamal wrote: I went back to this today. I am typing this from a scribbled sticky note in a big hurry - but i still believe I took the correct notes. It does seem there is no distinction between what ethernet advertises for flow control capability vs what it ends up negotiating with its partner i.e there is some ambiguity. I havent checked tg3, this on e1000 only. On Fri, 2006-07-07 at 08:28 -0400, jamal wrote: On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote: It's autonegotiated, check you kernel message logs when the link came up, you'll see this: tg3: eth0: Flow control is on for TX and on for RX. yikes - yes, this would be it. I could be wrong and i will double check: I think when the e1000 says via ethtool rx is on - it means that it is _advertising_ flow control as opposed to detecting partner has flow control capability. Auke, can you also check this as well? Semantic #1 For example, configure: ethtool -A eth0 rx off ethtool -A eth0 tx on debopolis:~# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: on The other side was set to do symmetric TX flow control only. Now enforce autonegotiation: ethtool -r eth0 ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off Ok, this is what i expected if this thing (output of ethtool) was supposed to store state as opposed to configuration. But if it is state that is stored, then what about that the values before autonegotian - surely that state is invalid, no? It would be nice (for debug/usability reasons) to be able to see what i configured vs what i end up negotiating with the link partner. I think this may be an ethtool issue, but it could also be a driver issue. I send 1 Mpps to eth0 and see no flow control packets back. good. So it does store state #2: The other semantic debopolis:~# ethtool -A eth0 rx on debopolis:~# ethtool -A eth0 tx on debopolis:~# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: on TX: on Other side was set to do symmetric TX flow control only. Now enforce autonegotiation: debopolis:~# ethtool -r eth0 lets see what we came up with: debopolis:~# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: on TX: on Now that is contradictory to #1 semantic - I would have expected this TX flow control on the e1000 to be off. Unless it is meant to store configuration info and not what you have negotiated. Trying sending traffic to the e1000 at about 1Mpps. I observe that the e1000 is sending out about 800Kpps of flow control packets back ;- So which semantics are correct? I would claim #2 flow control behavior to be a bug. I just dont have time to chase a fix - hopefully whoever reads this from the e1000 crowd can fix it. More importantly can we have two variables storing the two pieces on information separately instead of the ambiguity of just one? cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Bug in e1000 + semantics of flow control WAS(Re: [e1000]: flow control on by default - good idea really?
I went back to this today. I am typing this from a scribbled sticky note in a big hurry - but i still believe I took the correct notes. It does seem there is no distinction between what ethernet advertises for flow control capability vs what it ends up negotiating with its partner i.e there is some ambiguity. I havent checked tg3, this on e1000 only. On Fri, 2006-07-07 at 08:28 -0400, jamal wrote: On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote: It's autonegotiated, check you kernel message logs when the link came up, you'll see this: tg3: eth0: Flow control is on for TX and on for RX. yikes - yes, this would be it. I could be wrong and i will double check: I think when the e1000 says via ethtool rx is on - it means that it is _advertising_ flow control as opposed to detecting partner has flow control capability. Auke, can you also check this as well? Semantic #1 For example, configure: ethtool -A eth0 rx off ethtool -A eth0 tx on debopolis:~# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: on The other side was set to do symmetric TX flow control only. Now enforce autonegotiation: ethtool -r eth0 ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off Ok, this is what i expected if this thing (output of ethtool) was supposed to store state as opposed to configuration. But if it is state that is stored, then what about that the values before autonegotian - surely that state is invalid, no? It would be nice (for debug/usability reasons) to be able to see what i configured vs what i end up negotiating with the link partner. I think this may be an ethtool issue, but it could also be a driver issue. I send 1 Mpps to eth0 and see no flow control packets back. good. So it does store state #2: The other semantic debopolis:~# ethtool -A eth0 rx on debopolis:~# ethtool -A eth0 tx on debopolis:~# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: on TX: on Other side was set to do symmetric TX flow control only. Now enforce autonegotiation: debopolis:~# ethtool -r eth0 lets see what we came up with: debopolis:~# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: on TX: on Now that is contradictory to #1 semantic - I would have expected this TX flow control on the e1000 to be off. Unless it is meant to store configuration info and not what you have negotiated. Trying sending traffic to the e1000 at about 1Mpps. I observe that the e1000 is sending out about 800Kpps of flow control packets back ;- So which semantics are correct? I would claim #2 flow control behavior to be a bug. I just dont have time to chase a fix - hopefully whoever reads this from the e1000 crowd can fix it. More importantly can we have two variables storing the two pieces on information separately instead of the ambiguity of just one? cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
From: jamal [EMAIL PROTECTED] Date: Thu, 06 Jul 2006 09:03:38 -0400 On Wed, 2006-05-07 at 22:45 +0200, Krzysztof Oledzki wrote: On Wed, 5 Jul 2006, Auke Kok wrote: David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 15:20:39 -0400 BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow control has been on by default in the tg3 driver since the beginning, Are you sure about this Dave?;- because I do have a tg3 on my laptop. [EMAIL PROTECTED]:~/Desktop/maemo$ sudo ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off It's autonegotiated, check you kernel message logs when the link came up, you'll see this: tg3: eth0: Flow control is on for TX and on for RX. Take care. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote: It's autonegotiated, check you kernel message logs when the link came up, you'll see this: tg3: eth0: Flow control is on for TX and on for RX. yikes - yes, this would be it. I could be wrong and i will double check: I think when the e1000 says via ethtool rx is on - it means that it is _advertising_ flow control as opposed to detecting partner has flow control capability. Auke, can you also check this as well? cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Wed, 2006-05-07 at 22:45 +0200, Krzysztof Oledzki wrote: On Wed, 5 Jul 2006, Auke Kok wrote: David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 15:20:39 -0400 BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow control has been on by default in the tg3 driver since the beginning, Are you sure about this Dave?;- because I do have a tg3 on my laptop. [EMAIL PROTECTED]:~/Desktop/maemo$ sudo ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off [EMAIL PROTECTED]:~/Desktop/maemo$ uname -a Linux jzny2 2.6.16 #6 Fri Jun 9 15:29:40 EDT 2006 i686 GNU/Linux [EMAIL PROTECTED]:~/Desktop/maemo$ Maybe it is read from the eeprom and mine has it off? maybe e1000 only recently started to behave that way but it's the right thing to do IMHO. I will continue testing when i get proper hardware and post. And if only Robert and I are whining about this, we are adults and we can turn it off. I never tried it with large packets, so it may be helpful there for forwarding. Again, note that: It is consuming 10% (13-15% range) of my bandwidth. Granted that is at high speeds with small packets so may not be reflective of 96% of the world. But that would be 50kpps of my forwarding capacity being chewed unreasonably. So Auke, did you say performance was what people mostly bitched about? ;- As said earlier, e1000 always honors the EEPROM setting for this, which has been _on_ by default for all cards (AFAIK, that is). It has _never ever_ worked on e1000 for as long as i have used e1000. If it was intended to work, it must have been fixed in 2.6.16. So it is new behavior. I'm not sure: [EMAIL PROTECTED]:~# mii-tool -v eth0 Try ethtool -a eth0 to read and -A to set. cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
jamal wrote: On Wed, 2006-05-07 at 22:45 +0200, Krzysztof Oledzki wrote: On Wed, 5 Jul 2006, Auke Kok wrote: David Miller wrote: Flow control has been on by default in the tg3 driver since the beginning, Are you sure about this Dave?;- because I do have a tg3 on my laptop. [EMAIL PROTECTED]:~/Desktop/maemo$ sudo ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off mine says it's on :) Maybe it is read from the eeprom and mine has it off? Again, note that: It is consuming 10% (13-15% range) of my bandwidth. Granted that is at high speeds with small packets so may not be reflective of 96% of the world. But that would be 50kpps of my forwarding capacity being chewed unreasonably. So Auke, did you say performance was what people mostly bitched about? ;- yes, but that's linked with hardware that doesn't handle flowcontrol events properly, if you were doing large message TCP transfers over that you'd probably see even worse performance I bet (retransmits being dropped etc). Jesse is working on performance stuff, he'll gladly look into it :) As said earlier, e1000 always honors the EEPROM setting for this, which has been _on_ by default for all cards (AFAIK, that is). It has _never ever_ worked on e1000 for as long as i have used e1000. If it was intended to work, it must have been fixed in 2.6.16. So it is new behavior. Turns out that of the e1000 cards I can find around here that are plugged in actually are 50-50 distributed on/off, so I was wrong about it being on by default everywhere. Looking back through the code I see no changes affecting flow control setup as early as 2.6.12 ... There are some minor (new) HW changes but nothing that should have boken fc. Cheers, Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Thu, 2006-06-07 at 11:25 -0700, Auke Kok wrote: jamal wrote: [EMAIL PROTECTED]:~/Desktop/maemo$ sudo ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off mine says it's on :) Dell D610: [EMAIL PROTECTED]:~$ sudo lspci | grep -i bcm :02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 01) Whats yours? So ok, from what you say below i take it that even with the e1000s you didnt find consistency on different machines. Can you double check at least the kernels are = 2.6.16? I can tell you this hardware that caused me problems never had flow control working in the earlier kernels. Maybe it is read from the eeprom and mine has it off? Again, note that: It is consuming 10% (13-15% range) of my bandwidth. Granted that is at high speeds with small packets so may not be reflective of 96% of the world. But that would be 50kpps of my forwarding capacity being chewed unreasonably. So Auke, did you say performance was what people mostly bitched about? ;- yes, but that's linked with hardware that doesn't handle flowcontrol events properly, As i mentioned earlier: This is actually hardware that works with flow control ;- When it doesnt work - as in the case of the one i found advertising but not respecting flow control the bandwidth being consumed was 60% ;- To give you perspective, on average that is 500Mbps if you were doing large message TCP transfers over that you'd probably see even worse performance I bet (retransmits being dropped etc). I havent tested that, but it does seem unlikely. Jesse is working on performance stuff, he'll gladly look into it :) Jesse, if you want to reproduce this talk to me and i will give you a description of how to do it. Make sure you can record flow control packets both in send and receive. As said earlier, e1000 always honors the EEPROM setting for this, which has been _on_ by default for all cards (AFAIK, that is). It has _never ever_ worked on e1000 for as long as i have used e1000. If it was intended to work, it must have been fixed in 2.6.16. So it is new behavior. Turns out that of the e1000 cards I can find around here that are plugged in actually are 50-50 distributed on/off, so I was wrong about it being on by default everywhere. Looking back through the code I see no changes affecting flow control setup as early as 2.6.12 ... There are some minor (new) HW changes but nothing that should have boken fc. It could be something very basic. Since you have access to a variety of hardware (and variety of kernels i take it), you can go by elimination and find it. cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
Jamal wrote: Dell D610: [EMAIL PROTECTED]:~$ sudo lspci | grep -i bcm :02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 01) This is Broadcom tg3 hardware. Are we talking about e1000 or tg3? Flow control advertisement has always been on by default on tg3 and it doesn't depend on eeprom settings. Of course the link partner's advertisement settings can affect the final outcome of flow control settings on both ends. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
jamal wrote: On Tue, 2006-04-07 at 13:11 -0400, jamal wrote: I have a device connected to a e1000 that was erroneously advertising both tx/rx flow control but wasnt properly reacting to it. The default setup on the e1000 has rx flow control turned on. I was sending at wire rate gige from the device - which is about 1.48Mpps. The e1000 was in turn sending me flow control packets as per default/expected behavior. Unfortunately, it was sending a very large amount of packets. At one point i was seeing upto 1Mpps and on average, the flow control packets were consuming 60-70% of the bandwidth. Even when i fixed this behavior to act properly, allowing flow control on consumed up to 15% of the bandwidth. Clearly, this is a bad thing. Yes, the device in the first instance was at fault. But i have argued in the past that NAPI does just fine without flow control being turned on, so even chewing 5% of bandwidth on flow control is a bad thing.. As a compromise, can we declare flow control as an advanced feature and turn it off by default? People who feel it is valuable and know what they are doing can turn it off. I meant turn it on. BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow Control is using the EEPROM provided value, the module driver itself does not choose a default: e1000_param.c: /* User Specified Flow Control Override * * Valid Range: 0-3 * - 0 - No Flow Control * - 1 - Rx only, respond to PAUSE frames but do not generate them * - 2 - Tx only, generate PAUSE frames but ignore them on receive * - 3 - Full Flow Control Support * * Default Value: Read flow control settings from the EEPROM */ Turning flow control off usually (i.e. almost always) causes (significantly) _degraded_ performance. We should really leave it the way it is (as per eeprom setting), and this is best for most if not all people. The card itself has this value programmed, which makes it possible for the user to turn on/off flowcontrol per card consistently, which makes much more sense to me. Also considering e1000 hardware varies significantly. Moreover, most support calls here where people complain about degraded performance are about users who turned flow control _off_. :) Cheers, Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[e1000]: flow control on by default - good idea really?
jamal writes: The default setup on the e1000 has rx flow control turned on. I was sending at wire rate gige from the device - which is about 1.48Mpps. The e1000 was in turn sending me flow control packets as per default/expected behavior. Unfortunately, it was sending a very large amount of packets. At one point i was seeing upto 1Mpps and on average, the flow control packets were consuming 60-70% of the bandwidth. Even when i fixed this behavior to act properly, allowing flow control on consumed up to 15% of the bandwidth. Clearly, this is a bad thing. Yes, the device in the first instance was at fault. But i have argued in the past that NAPI does just fine without flow control being turned on, so even chewing 5% of bandwidth on flow control is a bad thing.. Interesting numbers. But one can argue that if there were no control packets there would be data packets sent and dropped instead. But I'll agree with you as the flow control seems hard to monitor and tune and takes debugging down to link and chiplevels. Much easier and robust just to read the dropped packet counter on the box that couldn't handle the load rather than following secret flow control packets somewhere else and try to figure out whats going on. As a compromise, can we declare flow control as an advanced feature and turn it off by default? People who feel it is valuable and know what they are doing can turn it off. At least for NAPI-scheme drivers. If you want more details just shoot. Cheers. --ro - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 13:11:56 -0400 Clearly, this is a bad thing. Yes, the device in the first instance was at fault. But i have argued in the past that NAPI does just fine without flow control being turned on, so even chewing 5% of bandwidth on flow control is a bad thing.. I think it should be on by default. If the machine is really busy, proper flow control keeps the sender from overflowing the RX ring of the receiver. Yes, with NAPI this happens less, but it does still happen. And I know that this is probably one of the least tested code paths in every single driver, RX ring emptying out, and in fact the Tigon2 and the earliest Tigon3 chips would lock up on you if the RX ring was emptied completely requiring a full chip reset when the watchdog timer fired. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 15:20:39 -0400 BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow control has been on by default in the tg3 driver since the beginning, maybe e1000 only recently started to behave that way but it's the right thing to do IMHO. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 15:20:39 -0400 BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow control has been on by default in the tg3 driver since the beginning, maybe e1000 only recently started to behave that way but it's the right thing to do IMHO. As said earlier, e1000 always honors the EEPROM setting for this, which has been _on_ by default for all cards (AFAIK, that is). Cheers, Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Wed, 5 Jul 2006, Auke Kok wrote: jamal wrote: On Tue, 2006-04-07 at 13:11 -0400, jamal wrote: I have a device connected to a e1000 that was erroneously advertising both tx/rx flow control but wasnt properly reacting to it. The default setup on the e1000 has rx flow control turned on. I was sending at wire rate gige from the device - which is about 1.48Mpps. The e1000 was in turn sending me flow control packets as per default/expected behavior. Unfortunately, it was sending a very large amount of packets. At one point i was seeing upto 1Mpps and on average, the flow control packets were consuming 60-70% of the bandwidth. Even when i fixed this behavior to act properly, allowing flow control on consumed up to 15% of the bandwidth. Clearly, this is a bad thing. Yes, the device in the first instance was at fault. But i have argued in the past that NAPI does just fine without flow control being turned on, so even chewing 5% of bandwidth on flow control is a bad thing.. As a compromise, can we declare flow control as an advanced feature and turn it off by default? People who feel it is valuable and know what they are doing can turn it off. I meant turn it on. BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow Control is using the EEPROM provided value, the module driver itself does not choose a default: e1000_param.c: /* User Specified Flow Control Override * * Valid Range: 0-3 * - 0 - No Flow Control * - 1 - Rx only, respond to PAUSE frames but do not generate them * - 2 - Tx only, generate PAUSE frames but ignore them on receive * - 3 - Full Flow Control Support * * Default Value: Read flow control settings from the EEPROM */ Turning flow control off usually (i.e. almost always) causes (significantly) _degraded_ performance. We should really leave it the way it is (as per eeprom setting), and this is best for most if not all people. The card itself has this value programmed, which makes it possible for the user to turn on/off flowcontrol per card consistently, which makes much more sense to me. Also considering e1000 hardware varies significantly. I was never able to find such tool for Linux or at least DOS. Where should I look for it? Best regards, Krzysztof Olędzki
Re: [e1000]: flow control on by default - good idea really?
On Wed, 5 Jul 2006, Auke Kok wrote: David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 15:20:39 -0400 BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow control has been on by default in the tg3 driver since the beginning, maybe e1000 only recently started to behave that way but it's the right thing to do IMHO. As said earlier, e1000 always honors the EEPROM setting for this, which has been _on_ by default for all cards (AFAIK, that is). I'm not sure: [EMAIL PROTECTED]:~# mii-tool -v eth0 eth0: negotiated 100baseTx-FD, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [EMAIL PROTECTED]:~# ethtool -d eth0|grep flow Receive flow control: disabled Transmit flow control: disabled [EMAIL PROTECTED]:~# uname -r 2.6.14.3 [EMAIL PROTECTED]:~# mii-tool -v eth0 eth0: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [EMAIL PROTECTED]:~# ethtool -d eth0|grep flow Receive flow control: enabled Transmit flow control: enabled [EMAIL PROTECTED]:~# uname -r 2.6.16.19 This is exactly the same hardware, only kernel was recently upgraded on the r2. Best regards, Krzysztof Olędzki
Re: [e1000]: flow control on by default - good idea really?
Krzysztof Oledzki wrote: On Wed, 5 Jul 2006, Auke Kok wrote: David Miller wrote: From: jamal [EMAIL PROTECTED] Date: Tue, 04 Jul 2006 15:20:39 -0400 BTW, As an addendum this default behavior changed around 2.6.16 it seems. Flow control has been on by default in the tg3 driver since the beginning, maybe e1000 only recently started to behave that way but it's the right thing to do IMHO. As said earlier, e1000 always honors the EEPROM setting for this, which has been _on_ by default for all cards (AFAIK, that is). I'm not sure: [EMAIL PROTECTED]:~# ethtool -d eth0|grep flow Receive flow control: disabled Transmit flow control: disabled [EMAIL PROTECTED]:~# ethtool -d eth0|grep flow Receive flow control: enabled Transmit flow control: enabled Same physical NIC or different ones? In the latter case your NICs may have different EEPROM settings. You can dump the EEPROM and compare, do something like `ethtool -e eth0 raw on length 64| hexdump` for both interfaces and send it to me. Only the first 64 bytes should matter. Cheers, Auke - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[e1000]: flow control on by default - good idea really?
CCing anybody who may have stakes on this. Ignore the email if this doesnt interest you. Ok, folks - i had deferred this discussion but it bit me in the ass. I just spend an hour debugging it (and in the process blew up a gbic i borrowed, so my day aint going well since i actually have to pay for this and cant really do the testing i was planning to;-). I have a device connected to a e1000 that was erroneously advertising both tx/rx flow control but wasnt properly reacting to it. The default setup on the e1000 has rx flow control turned on. I was sending at wire rate gige from the device - which is about 1.48Mpps. The e1000 was in turn sending me flow control packets as per default/expected behavior. Unfortunately, it was sending a very large amount of packets. At one point i was seeing upto 1Mpps and on average, the flow control packets were consuming 60-70% of the bandwidth. Even when i fixed this behavior to act properly, allowing flow control on consumed up to 15% of the bandwidth. Clearly, this is a bad thing. Yes, the device in the first instance was at fault. But i have argued in the past that NAPI does just fine without flow control being turned on, so even chewing 5% of bandwidth on flow control is a bad thing.. As a compromise, can we declare flow control as an advanced feature and turn it off by default? People who feel it is valuable and know what they are doing can turn it off. If you want more details just shoot. cheers, jamal PS:- BTW, even turning off flow control on e1000 didnt give as good performance as in the old days on this machine - but i dont want to go into that discussion. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [e1000]: flow control on by default - good idea really?
On Tue, 2006-04-07 at 13:11 -0400, jamal wrote: CCing anybody who may have stakes on this. Ignore the email if this doesnt interest you. Ok, folks - i had deferred this discussion but it bit me in the ass. I just spend an hour debugging it (and in the process blew up a gbic i borrowed, so my day aint going well since i actually have to pay for this and cant really do the testing i was planning to;-). I have a device connected to a e1000 that was erroneously advertising both tx/rx flow control but wasnt properly reacting to it. The default setup on the e1000 has rx flow control turned on. I was sending at wire rate gige from the device - which is about 1.48Mpps. The e1000 was in turn sending me flow control packets as per default/expected behavior. Unfortunately, it was sending a very large amount of packets. At one point i was seeing upto 1Mpps and on average, the flow control packets were consuming 60-70% of the bandwidth. Even when i fixed this behavior to act properly, allowing flow control on consumed up to 15% of the bandwidth. Clearly, this is a bad thing. Yes, the device in the first instance was at fault. But i have argued in the past that NAPI does just fine without flow control being turned on, so even chewing 5% of bandwidth on flow control is a bad thing.. As a compromise, can we declare flow control as an advanced feature and turn it off by default? People who feel it is valuable and know what they are doing can turn it off. I meant turn it on. BTW, As an addendum this default behavior changed around 2.6.16 it seems. cheers, jamal - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
