RE: Linux shaping packet loss
> Autoneg is a required part of the gig E specification so you'd only be > causing yourself trouble by turning it off. (I don't know if > it'll also break automatic MDI/MDI-X (crossover) configuration, for > an example of something that's nice to have.) At least on 450x series enhanced linecards, disabling autonegotiation disables auto MDI/MDI-X. You have to enable autonegotiation but you can provide specified offered and acceptable parameters, eg: "speed auto 100" to enable autonegotiation but only allow negotiation of 100 mb. (speed auto 10 100 / speed auto 1000 / etc) -- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org
Re: Linux shaping packet loss
> What's good for really cheap gigabit, redundant, high throughput Well .. I'd say you could pick any two of those and come up with a list .. but we use Packeteer (now owned by Bluecoat) to great success. It fails the first requirement miserably, IMHO, though. I've also used these in a MDU setting, but certainly not at gigabit speeds : http://www.netequalizer.com/nda.htm They claim models are available up to 5gbps ($11k). 1gbps is ~$9k. Cheers, Michael Holstein Cleveland State University
Re: Linux shaping packet loss
Thanks to all that replied. Trial and error it is ... I'm now waiting (22 hours later) for it to break again after I changed the priority on the "default" catch-all class. It lasted five days before. I'm looking at CBQ but it's not at all friendly relative to HTB. If I'm forced to go down the proprietary traffic-shaping route. What's good for really cheap gigabit, redundant, high throughput (including during 64-byte UDP attacks) shapers ? Suggestions appreciated. Chris 2009/12/9 Nickola Kolev > На Wed, 09 Dec 2009 06:38:31 + > gordon b slater написа: > > > On Wed, 2009-12-09 at 08:02 +0200, Bazy wrote: > > > > > Hi Chris, > > > > > > Try setting txqueuelen to 1000 on the interfaces and see if you > > > still get a lot of packet loss. > > > > > > > Yes, good point and well worth a try. Rereading Chris's post about > > "250Mbps" and "forty queues", the "egress" could well be bumping the > > end of a default fifo line. > > > > If 1000 is too high for your kit try pushing it upwards gradually from > > the default of 100 (?) but back off if you get drops or strangeness in > > ifconfig output on the egress i/f. > > The default *is* 1000. From the ifconfig man page: > > txqueuelen length > > Set the length of the transmit queue of the device. It is useful to > set this to small values for slower devices with a high atency (modem > links, ISDN) to prevent fast bulk transfers from disturbing interactive > traffic like telnet too much. > > So, if you should touch it if and only if you want to have (supposedly) > finer grained control on queueing, as the hardware device also does > some reordering before it puts the data on the wire. > > > I append grep-ped ifconfig outputs into a file every hour on a cron > > job until I'm happy that strangeness doesn't happen, they never do > > when you're watching sadly. > > > > TC problems aren't always about the TC itself, the physical interfaces > > are inherently part of the "system", as my long rambling 5am+ > > up-all-night-over-ssh post about reseating NICs was trying to hint > > at. > > > > Nice one Bazy > > > > Gord > > > > > > > > > > > > > -- > Regards, > Nickola > >
Re: Linux shaping packet loss
На Wed, 09 Dec 2009 06:38:31 + gordon b slater написа: > On Wed, 2009-12-09 at 08:02 +0200, Bazy wrote: > > > Hi Chris, > > > > Try setting txqueuelen to 1000 on the interfaces and see if you > > still get a lot of packet loss. > > > > Yes, good point and well worth a try. Rereading Chris's post about > "250Mbps" and "forty queues", the "egress" could well be bumping the > end of a default fifo line. > > If 1000 is too high for your kit try pushing it upwards gradually from > the default of 100 (?) but back off if you get drops or strangeness in > ifconfig output on the egress i/f. The default *is* 1000. From the ifconfig man page: txqueuelen length Set the length of the transmit queue of the device. It is useful to set this to small values for slower devices with a high atency (modem links, ISDN) to prevent fast bulk transfers from disturbing interactive traffic like telnet too much. So, if you should touch it if and only if you want to have (supposedly) finer grained control on queueing, as the hardware device also does some reordering before it puts the data on the wire. > I append grep-ped ifconfig outputs into a file every hour on a cron > job until I'm happy that strangeness doesn't happen, they never do > when you're watching sadly. > > TC problems aren't always about the TC itself, the physical interfaces > are inherently part of the "system", as my long rambling 5am+ > up-all-night-over-ssh post about reseating NICs was trying to hint > at. > > Nice one Bazy > > Gord > > > > > -- Regards, Nickola
Re: Linux shaping packet loss
On Wed, 2009-12-09 at 06:38 +, gordon b slater wrote: > If 1000 is too high for your kit try pushing it upwards gradually from > the default of 100 meh! 6am+insomniac blues for a Gigeth it's more likely to be 1000 already, so push it up to 1 in stages - you get the idea.
Re: Linux shaping packet loss
On Wed, 2009-12-09 at 08:02 +0200, Bazy wrote: > Hi Chris, > > Try setting txqueuelen to 1000 on the interfaces and see if you still > get a lot of packet loss. > Yes, good point and well worth a try. Rereading Chris's post about "250Mbps" and "forty queues", the "egress" could well be bumping the end of a default fifo line. If 1000 is too high for your kit try pushing it upwards gradually from the default of 100 (?) but back off if you get drops or strangeness in ifconfig output on the egress i/f. I append grep-ped ifconfig outputs into a file every hour on a cron job until I'm happy that strangeness doesn't happen, they never do when you're watching sadly. TC problems aren't always about the TC itself, the physical interfaces are inherently part of the "system", as my long rambling 5am+ up-all-night-over-ssh post about reseating NICs was trying to hint at. Nice one Bazy Gord
Re: Linux shaping packet loss
On Tue, Dec 8, 2009 at 5:14 PM, Chris wrote: > Thanks, Steiner and everyone for the input. It's good to see the list is > still as friendly as ever. > > There are two paths I'm trying to get my head round after someone offlist > helpfully suggested putting cburst and burst on all classes. > > My thoughts are that any dropped packets on the parent class is a bad thing: > > qdisc htb 1: root r2q 10 default 265 direct_packets_stat 448 ver 3.17 > Sent 4652558768 bytes 5125175 pkt (dropped 819, overlimits 10048800 > requeues 0) > rate 0bit 0pps backlog 0b 28p requeues 0 > > Until now I've had Rate and Ceil at the same values on all the classes but I > take the point about cburst and burst allowing greater levels of borrowing > so I've halved the Rate for all classes and left the Ceil the same. > > I've gone done this route mainly because I really can't risk breaking things > with incorrect cburst and burst values (if anyone can please tell me on an > i686 box at, say, 10Mbps the ideal values I can translate them into higher > classes, TC seems to work them out as 1600b/8 mpu by default and the timing > resolution confuses me.) > > Thanks again, > > Chris > > 2009/12/8 > >> > Won't say I'm an expert with TC, but anytime I see packet loss on an >> > interface I always check the interface itself...10% packet loss is >> > pretty much what you would get if there was a duplex problem. I always >> > try to hard set my interfaces on both the Linux machines and Switches. >> >> Used to set everything hard five years ago. Nowadays auto works just >> fine most of the time. >> >> Steinar Haug, Nethelp consulting, sth...@nethelp.no >> >> > Hi Chris, Try setting txqueuelen to 1000 on the interfaces and see if you still get a lot of packet loss. Bazy
Re: Linux shaping packet loss
Apologies to all on handheld devices. If you're not into BSD or Linux TC operationally, skip this post. Due to my usual rambling narrative style for "alternative" troubleshooting I was going to mail this direct to the OP but I was persuaded AMBJ by a co-conspirator to post this to list in full. # @all with similar "traffic shaping" problems Googling in the future: On Wed, 2009-12-09 at 12:07 +1100, Simon Horman wrote: > but trying to use much > more than 90% of the link capacity ..though not directly relevant in this case, for lower speed links and things like xDSL to the CPE that 90% must include protocol overheads (you are getting close to bone in that last 10%) and _much_ more affective (<- that's A-ffective) things like actual modem "sync speed". It depends how the TC is calc'ed/applied of course. Just a general note for a more CPE-oriented occurence of this. So kids, if you're struggling with your IPCOP in a SOHO shop with ADSL+PPPoE, this means you! Meanwhile, back at our level... @all generally: do many of us use Linux TC at small-carrier level? I know of a lot of BSD boxen out there that handle huge complex flows but I suspect Linux kernel is less popular for this - or am I assuming wrong? Personally I'd lean to BSD for big stuff and Linux on for CPE, am I out of touch nowadays? Fully back on topic from here on... @Chris - I've not used RED in any anger, sorry. Other than a typo in the config for the affected queue (maybe an extra digit loose somewhere?), things are definitely going to get complicated. Is something exceeding a tc bucket mtu occasionally? Chris wrote: > >My thoughts are that any dropped packets on the parent class is a bad >thing: yes, generally speaking, but. > >qdisc htb 1: root r2q 10 default 265 direct_packets_stat 448 ver 3.17 > Sent 4652558768 bytes 5125175 pkt (dropped 819, overlimits 10048800 >requeues 0) > rate 0bit 0pps backlog 0b 28p requeues 0 ... in the above example, that loss rate is extremely low at 000.0159% ( 819 / 5125175 %) It may not be a representative sample, but I just thought I'd check you hadn't dropped a few significant digits in a %loss calc along the way :) That level of loss if operationally insignificant of course, especially for TCP. As you are I'm sure aware, perfect TC through any box is pretty specialist and usually unique to that placement. Without any graphical output, queues and the like are extremely difficult to visualize (mentally) under load (though for smaller boxes the RRD graphs in pfSENSE are nicely readable - see below). Because of this I usually try to eliminate ~everything~ else before I get into qdisks and the nitty-gritty. As a natural control fr/geek I've wasted far to many hours stuck in the buckets to no real improvement in many cases. Chris wrote: > I've isolated it to the egress HTB qdisc > good, though read on for a strange tale You MUST make a distinction between TC dropping the packets and the interface dropping the packets; I see in your later post a TC qdisc line showing that tc itself had dropped packets, BUT it ALWAYS pays to check at the same time (using ifconfig) that no packets are reported being dropped by the interfaces as well. I've had 2 or 3 occasions where `TC drops` were actually somehow linked to _interface_ drops and it really threw me, we never did work out why. The interaction confounded us totally. IF the INTERFACES are ALSO dropping in ifconfig, THEN, and ONLY then, you are into the lowest layer. So, with that in mind and the sheer complexity of possibilities, here's how I personally approach difficult BSD/Linux "TC problems". Note that I have zero experience or inclination towards Cisco TC: Kick the tyres! A lot of people mentioned layer 2 link-config problems, but as far as I can see, no-one has suggested quickly yanking the cables and blowing the dust off the the ends. Whenever I have to reach for a calculator or pen for a problem, I first swap out the interconnects to reduce the mental smoke ;) Next, I check the NICs to see if they're unseated (if applicable), or CPU (think: rogue process - use top) or even bus utililisation if you have only 32bit PCI NICs in a busy box. Next. does the box do anything else like Snort/Squid/etc at the same time? To eliminate wierdness and speedup troubleshooing if TC is acting strange I'd run tcpdump continually from the very start of my troubleshooting, dumping into small 10MB-ish files - use the special -C option ="split to filesize" and the -W option to set about 100 files in a ring buffer so that you have a decent history to go back through if you need it, without clogging the fisystem of the box with TB or packetdata :) (splitting them into 10MB files at the start leads to fast analysis in the shark, though you could carve up larger files manually I guess) That way, if the TC hurts your brain run the dumps them through wireshark's "expert info" filter while you have a coffee. (Analylse>ExpertInfo I
Re: Linux shaping packet loss
On Tue, Dec 08, 2009 at 03:14:01PM +, Chris wrote: > Thanks, Steiner and everyone for the input. It's good to see the list is > still as friendly as ever. > > There are two paths I'm trying to get my head round after someone offlist > helpfully suggested putting cburst and burst on all classes. > > My thoughts are that any dropped packets on the parent class is a bad thing: > > qdisc htb 1: root r2q 10 default 265 direct_packets_stat 448 ver 3.17 > Sent 4652558768 bytes 5125175 pkt (dropped 819, overlimits 10048800 > requeues 0) > rate 0bit 0pps backlog 0b 28p requeues 0 > > Until now I've had Rate and Ceil at the same values on all the classes but I > take the point about cburst and burst allowing greater levels of borrowing > so I've halved the Rate for all classes and left the Ceil the same. > > I've gone done this route mainly because I really can't risk breaking things > with incorrect cburst and burst values (if anyone can please tell me on an > i686 box at, say, 10Mbps the ideal values I can translate them into higher > classes, TC seems to work them out as 1600b/8 mpu by default and the timing > resolution confuses me.) Silly question, but are you leaving some headroom? Its a little while since I've worked with HTB and from my experience the exact results do depend somewhat on the kernel that is in use, but trying to use much more than 90% of the link capacity caused troubles for me. In particular I'm referring to the ceil value of the root class. I also noticed that at higher packet rates (I was doing gigabit in a lab) that increasing r2q helped me. However I was looking at (UDP) throughput not packet loss.
Re: Linux shaping packet loss
On 9/12/2009, at 4:47 AM, Tony Finch wrote: Autoneg is a required part of the gig E specification so you'd only be causing yourself trouble by turning it off. (I don't know if it'll also break automatic MDI/MDI-X (crossover) configuration, for an example of something that's nice to have.) Yes it will break auto MDI/MDI-X. -- Nathan Ward
Re: Linux shaping packet loss
On Tue, 8 Dec 2009 13:13:03 + Chris wrote: > Hi All, > > It would be appreciated if anyone using TC on Linux for shaping could please > help with an intermittent problem on an egress interface. Well, it's unbelievable, but almost 5 hours and 11 mails later not even one of them has mentioned something different than L2 incompatibilities! And this, IMHO, has nothing to do with Chris problems. I'd really expect more from the guys that make the Internet run... Anyway... :) > I'm seeing about ten per cent of packet loss for all classes at seemingly > quiet times and random parts of the day using about forty classes and > 250Mbps. I've isolated it to the egress HTB qdisc. I'd start with a careful revisit of each class and the classifier that goes with it. I'd pay special attention to go with u32/hash classifiers (filters), and not with iptables. You can try to visualize the number of packets in each class (queued,dropped), and that way you will probably could see where the problem is. > Any TC experts out there have a spare minute please ? Any thoughts on the > RED qdisc ? As for this, I'd suggest to take a look at: [1] http://lartc.org/howto/lartc.adv-qdisc.red.html [2] http://www.opalsoft.net/qos/DS-26.htm > Thanks very much, > > Chris -- Best regards, Nickola
Re: Linux shaping packet loss
On Tue, Dec 8, 2009 at 7:18 AM, Matlock, Kenneth L wrote: > These days at 1Gb+ Full-Duplex seems to be the 'default' for > auto-negotiation failures. > Thankfully it's even more than a "seems to be" - it's written into the IEEE spec that if duplex negotiation fails then the default is full duplex for 1Gbps, as opposed to HDX for 100Mbps and earlier. Scott
Re: Linux shaping packet loss
> From my own experience, turning off auto negotiate can lead to unusual > behavior later on I too had this crop up in an unusual manner .. the hardware was HP with Intel Pro 1000 on one side, and Cisco 65xx on the other. Neither side saw errors, and (most) everything seemed to work .. however, one java app that depended on SSL would constantly fail when it tried to retrieve a file. Both sides hard coded (didn't matter to what) wouldn't work. When we upgraded to GigE blades, it still wouldn't work in any hard-coded configuration, even though the O/S (Win2k3 .. RDP, FTP, etc.) appeared to work. Set both sides auto/auto : bam. problem solved. The app was Sonicwall Email Security (on the off-chance someone else is fighting that same issue). Cheers, Michael Holstein Cleveland State University
Re: Linux shaping packet loss
On 12/8/09 8:13 AM, Joe Abley wrote: I've also heard people say that whatever you think about autoneg in Fast Ethernet, on Gigabit and 10GE interfaces it's pretty much never the right idea to turn autoneg off. From my own experience, turning off auto negotiate can lead to unusual behavior later on that may cause a bit of grief. Our SAN (LHN NSM260) refused flat out to do 802.3ad - giving us duplex errors. Took us around an hour of diagnosing - first we thought it was the switch, then we thought it was the cables we used, etc. Finally it dawned on me that my partner is notorious for hard coding ports on our own equipment. Low and behold, after her swearing up and down that there's no way its that, we set both ends to auto negotiate and boom, bonding came up happy as a clam. Only one port on our entire setup that is hard coded - 10BaseT-FD - and thats only because the darn thing refuses to auto negotiate to full duplex for 10BaseT links. I'm almost positive that a year or two down the line, we're going to forget that is there when we change the link to 100BaseT. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org/ http://www.ahbl.org
Re: Linux shaping packet loss
> The biggest problem with duplex had to do with 100mb. > > Cisco (and a lot of other companies) decided in their infinite wisdom > that at 100mb if auto-negotiation fails, to use half duplex as the > default. No, that wasn't those companies deciding to do so in their infinite wisdom. That was those companies deciding to follow the IEEE standard! Cisco and others may be to blame for a lot of things, but not this one. Steinar Haug, Nethelp consulting, sth...@nethelp.no
Re: Linux shaping packet loss
On Tue, 8 Dec 2009, Joe Abley wrote: > > I find there is a lot of hard-coded wisdom that hard-coded speed duplex > are the way to avoid pain. That was definitely true in the mid-to-late 1990s. > The last time I saw anybody do a modern survey of switches, routers and > hosts, however, it seemed like the early interop problems with autoneg > on FE really don't exist today, and on balance there are probably more > duplex problems caused by hard-configured ports that are poorly > maintained in the heat of battle than there are because autoneg is > flaky. Yes. The autoneg specification was fixed in 1998 so modern kit should interoperate properly. > I've also heard people say that whatever you think about autoneg in Fast > Ethernet, on Gigabit and 10GE interfaces it's pretty much never the > right idea to turn autoneg off. Autoneg is a required part of the gig E specification so you'd only be causing yourself trouble by turning it off. (I don't know if it'll also break automatic MDI/MDI-X (crossover) configuration, for an example of something that's nice to have.) Tony. -- f.anthony.n.finchhttp://dotat.at/ GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS. MODERATE OR GOOD.
RE: Linux shaping packet loss
The biggest problem with duplex had to do with 100mb. Cisco (and a lot of other companies) decided in their infinite wisdom that at 100mb if auto-negotiation fails, to use half duplex as the default. So if you have both sides at auto, or both sides hard-set it's all good. But if one side is hard-set and the other is auto, a lot of times the auto device will come up 100/Half. These days at 1Gb+ Full-Duplex seems to be the 'default' for auto-negotiation failures. Ken Matlock Network Analyst Exempla Healthcare (303) 467-4671 matlo...@exempla.org -Original Message- From: Joe Abley [mailto:jab...@hopcount.ca] Sent: Tuesday, December 08, 2009 8:14 AM To: sth...@nethelp.no Cc: nanog@nanog.org Subject: Re: Linux shaping packet loss On 2009-12-08, at 15:01, sth...@nethelp.no wrote: >> Won't say I'm an expert with TC, but anytime I see packet loss on an >> interface I always check the interface itself...10% packet loss is >> pretty much what you would get if there was a duplex problem. I always >> try to hard set my interfaces on both the Linux machines and Switches. > > Used to set everything hard five years ago. Nowadays auto works just > fine most of the time. I find there is a lot of hard-coded wisdom that hard-coded speed duplex are the way to avoid pain. The last time I saw anybody do a modern survey of switches, routers and hosts, however, it seemed like the early interop problems with autoneg on FE really don't exist today, and on balance there are probably more duplex problems caused by hard-configured ports that are poorly maintained in the heat of battle than there are because autoneg is flaky. I've also heard people say that whatever you think about autoneg in Fast Ethernet, on Gigabit and 10GE interfaces it's pretty much never the right idea to turn autoneg off. I am profoundly ignorant of the details of layer-2. It'd be nice to have more than vague rhetoric to guide me when configuring interfaces. What reliable guidance exists for this stuff? Joe
Re: Linux shaping packet loss
Thanks, Steiner and everyone for the input. It's good to see the list is still as friendly as ever. There are two paths I'm trying to get my head round after someone offlist helpfully suggested putting cburst and burst on all classes. My thoughts are that any dropped packets on the parent class is a bad thing: qdisc htb 1: root r2q 10 default 265 direct_packets_stat 448 ver 3.17 Sent 4652558768 bytes 5125175 pkt (dropped 819, overlimits 10048800 requeues 0) rate 0bit 0pps backlog 0b 28p requeues 0 Until now I've had Rate and Ceil at the same values on all the classes but I take the point about cburst and burst allowing greater levels of borrowing so I've halved the Rate for all classes and left the Ceil the same. I've gone done this route mainly because I really can't risk breaking things with incorrect cburst and burst values (if anyone can please tell me on an i686 box at, say, 10Mbps the ideal values I can translate them into higher classes, TC seems to work them out as 1600b/8 mpu by default and the timing resolution confuses me.) Thanks again, Chris 2009/12/8 > > Won't say I'm an expert with TC, but anytime I see packet loss on an > > interface I always check the interface itself...10% packet loss is > > pretty much what you would get if there was a duplex problem. I always > > try to hard set my interfaces on both the Linux machines and Switches. > > Used to set everything hard five years ago. Nowadays auto works just > fine most of the time. > > Steinar Haug, Nethelp consulting, sth...@nethelp.no > >
Re: Linux shaping packet loss
On 2009-12-08, at 15:01, sth...@nethelp.no wrote: >> Won't say I'm an expert with TC, but anytime I see packet loss on an >> interface I always check the interface itself...10% packet loss is >> pretty much what you would get if there was a duplex problem. I always >> try to hard set my interfaces on both the Linux machines and Switches. > > Used to set everything hard five years ago. Nowadays auto works just > fine most of the time. I find there is a lot of hard-coded wisdom that hard-coded speed duplex are the way to avoid pain. The last time I saw anybody do a modern survey of switches, routers and hosts, however, it seemed like the early interop problems with autoneg on FE really don't exist today, and on balance there are probably more duplex problems caused by hard-configured ports that are poorly maintained in the heat of battle than there are because autoneg is flaky. I've also heard people say that whatever you think about autoneg in Fast Ethernet, on Gigabit and 10GE interfaces it's pretty much never the right idea to turn autoneg off. I am profoundly ignorant of the details of layer-2. It'd be nice to have more than vague rhetoric to guide me when configuring interfaces. What reliable guidance exists for this stuff? Joe
Re: Linux shaping packet loss
> Won't say I'm an expert with TC, but anytime I see packet loss on an > interface I always check the interface itself...10% packet loss is > pretty much what you would get if there was a duplex problem. I always > try to hard set my interfaces on both the Linux machines and Switches. Used to set everything hard five years ago. Nowadays auto works just fine most of the time. Steinar Haug, Nethelp consulting, sth...@nethelp.no
Re: Linux shaping packet loss
Won't say I'm an expert with TC, but anytime I see packet loss on an interface I always check the interface itself...10% packet loss is pretty much what you would get if there was a duplex problem. I always try to hard set my interfaces on both the Linux machines and Switches. Bret Chris wrote: Hi All, It would be appreciated if anyone using TC on Linux for shaping could please help with an intermittent problem on an egress interface. I'm seeing about ten per cent of packet loss for all classes at seemingly quiet times and random parts of the day using about forty classes and 250Mbps. I've isolated it to the egress HTB qdisc. Any TC experts out there have a spare minute please ? Any thoughts on the RED qdisc ? Thanks very much, Chris
