Was it Mark who suggested bringing this to NANOG? I like that. Maybe anyone who sees this should track down who is sourcing the traffic and then follow up with the customer to find out what app or service they were using. Post to this list. When enough information is collected, someone who participates in NANOG brings it up there. If that doesn’t work, maybe try publicly shaming the service. If Netflix can shame ISPs for being slow, ISPs should be able to shame edge providers for aggressive TCP acceleration techniques that push other traffic aside.
Speaking of which, it doesn’t address this exact topic, but Sandvine had an interesting blog post (somewhat dated) about streaming traffic and its tendency to push regular web traffic aside, and the commercial incentives for “selfish” behavior. It should be mandatory reading for any regulators who the technical issues are simple, and the incentives are all stacked on the side of the mean old ISPs trying to throttle the virtuous content and edge providers. https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/exposing-the-technical-and-commercial-factors-underlying-internet-quality-of-experience.pdf From: George Skorup (Cyber Broadcasting) Sent: Tuesday, April 28, 2015 12:06 PM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. The other thing that really sucks about this is the SM is almost unmanageable with the VC being so overloaded. In this condition, you can use the AP LUID proxy to get into the SM just fine, but that's not really what I care about, dropped SNMP to/from the SM is the bigger issue. I suppose you could use the AP's SNMP proxy, but then you'd have to know the LUID, which changes on reboot, and is just not optimal. I wonder if Cambium can go back to prioritizing management in a future release. My memory is foggy, but I think this was hurting HP traffic a long, long time ago. On 4/28/2015 8:02 AM, Kurt Fankhauser wrote: Thanks for all the information about this that everyone shared, I now know that I'm not crazy. Although it sounds like we may have this as more of a problem in the future.... Kurt Fankhauser Wavelinc Communications P.O. Box 126 Bucyrus, OH 44820 http://www.wavelinc.com tel. 419-562-6405 fax. 419-617-0110 On Mon, Apr 27, 2015 at 11:39 AM, Ken Hohhof <[email protected]> wrote: I’ve spotted exactly as George describes, torch at our border router sees way more traffic than I see at the SM or the customer’s router. I think one time I traced it to a Limelight Networks IP, but I didn’t write it down so I’m relying on memory so I could well be wrong. From: George Skorup (Cyber Broadcasting) Sent: Monday, April 27, 2015 10:29 AM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. Yeah, it's almost always streaming video. Except the one guy I found running the "download manager" on his PC was doing USENET crap, probably porn, movies, music, etc. It's fairly easy to see what's going on. For instance, a customer is on 6x1 Canopy. There's 6Mbps RF downlink and 12Mbps Rx at AP's ethernet interface. This is all fine and dandy with Canopy because the AP controls the downlink traffic and doesn't put more on it than the QoS allows. Like Kurt, the first time I saw this, I was running the MT torch tool and thought the Canopy QoS was broken, too. Now, for UBNT or really anything else where the CPE does the limiting, the AP is sending all that traffic to the CPE. So now you have to resort to policing at an upstream router which I'd rather not do to keep the load off of the little MT routers. Even still, after you do that, all that extra traffic is still coming in on the backhaul, so all you've done is moved the problem, which helps relieve the AP stress, but it still sucks to have to take on double the traffic. You can move the limiting all the way to your border(s), but that extra traffic is still taking up bandwidth somewhere. On 4/27/2015 9:48 AM, Chuck McCown wrote: I haven’t been following this thread closely, but has anyone identified the traffic? From: Ken Hohhof Sent: Monday, April 27, 2015 8:46 AM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. If we could get Procera to create a signature for this type of behavior, then we could create a rule to restrict this traffic to some percentage less than 50% of the subscriber’s speed tier. This would perhaps have several desirable results: - if the CDN algorithm aims to oversubscribe the customer’s pipe by 2X, then making the pipe look 0.5X as small should counteract that - I would actually set it lower than 50%, the objective to have the customer say “XYZ service sucks” and complain to them or stop using them, rather than “my Internet sucks” - if we can pool our information about who is doing this (not just the CDN but the content provider paying them), we could complain directly to them, and if necessary take the position that blocking or throttling their traffic would be reasonable network management and allowable under net neutrality rules, since they are in effect attacking our network with a flood of traffic beyond what the customer has subscribed to Another approach would be to rate limit this traffic with a queue that has a very large buffer, introducing latency rather than packet loss, hoping that their algorithm recognizes late ACKs as a sign of congestion and will back off the sending rate. But when I asked Simon about buffer size in Procera, my understanding was that it’s more of a traffic policing box than a shaping box. From: Wireless Admin Sent: Monday, April 27, 2015 9:08 AM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. Ken, Your assessment of the problem is exactly correct. I was going to compare it tor DoS as you did here. I don’t see an easy fix for this. Steve -------------------------------------------------------------------------- From: Af [mailto:[email protected]] On Behalf Of Ken Hohhof Sent: Monday, April 27, 2015 10:00 AM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. I don’t think you’re understanding the situation that we are speculating is happening. He is using Cambium QoS. However, he is seeing twice that amount of traffic destined to the customer, the SM is throwing half of it away (as it should), and as a result the customer’s service sucks. The problem is that the sender is not observing traditional congestion control, it is not backing off the sending rate when it sees high packet loss. That’s why I say it is similar to a DoS attack, someone sending far more traffic than the subscriber can receive. From: David Milholen Sent: Monday, April 27, 2015 7:05 AM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. This is why we like setting the QOS at the subscriber. If I remember the cambium burst allocation ignores tcp and udp and work strictly on a token bit system. We do not receive these complaints. The only time I hear them is if we get overloaded at the backhaul link. On 4/26/2015 8:58 PM, Ken Hohhof wrote: I think George forgot the sarcasm emoticon. Also note that the problem here is the edge provider is sending more than the customer’s plan rate, ignoring TCP congestion control. Not only does this consume Internet bandwidth over and above what the customer has subscribed to, it makes anything else the customer is trying to do on the Internet unusable because normal TCP is unusable with 50% packet loss. It is not surprising the customer calls saying his Internet is slow. There’s a saying that comes to mind, involving a 5 pound bag. From: Faisal Imtiaz Sent: Sunday, April 26, 2015 8:37 PM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. I see that the net neutrality is going to be the next boogieman under the bed for WISP's from now on... Please, please, please, correct your understanding on Net-Neutrality... It allows for one to traffic shape any and all kinds of traffic, as long as :- a) You declare your practice on your website. b) You DON"T DO IT specific to A SPECIFIC Network.. i.e. all VOIP, or all Video, or ALL Streaming.. (applying a throttle on video to netflix while allowing Hulu would be considered a violation, but applying throttle to all types of video content is NOT !) :) Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: [email protected] ------------------------------------------------------------------------ From: "George Skorup (Cyber Broadcasting)" mailto:[email protected] To: [email protected] Sent: Sunday, April 26, 2015 7:12:13 PM Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. So you'd be purposely slowing down or blocking legitimate traffic from an edge provider to the customer? Oh no, net neutrality violation! So when everyone starts with the 4k streaming and we're selling the customer 20Mbps, then we have to take on 40Mbps because of this!? On 4/26/2015 5:58 PM, Ken Hohhof wrote: I could justify declaring such traffic an attack and blocking the source as malicious. From: George Skorup (Cyber Broadcasting) Sent: Sunday, April 26, 2015 4:30 PM To: [email protected] Subject: Re: [AFMUG] 450SM sustain bucket throttle not working.. Yep, I see this all the time and Ken is exactly right. The Canopy QoS works exactly as designed, the AP is definitely not delivering more than the sustained rate, but is instead discarding the extra 50%. I've tested this situation thoroughly. Stick a MT simple queue in at the upstream router and the 2X rate traffic stops hitting the AP's ethernet interface, but it's still coming in at double the sustained rate farther upstream. There's no way around it except throwing bandwidth at it. This is CDN traffic. And when the customer thinks they can install one of those "internet download managers" to speed up their connection. The only thing it does is screw with TCP acks or window sizes or something which just puts more traffic on your transit just to be discarded at the congestion point (SM, queue, Procera, whatever). Gotta love it. You'd think with 70% of the internets being streaming video they'd think hmm.. maybe we can cut down on the peering congestion by NOT doing this crap. But no. On 4/26/2015 11:01 AM, Ken Hohhof wrote: Sorry to answer a question with a question, but are you measuring at the SM, or at some upstream router? The reason I ask, is I have seen some CDN traffic that does not seem to follow traditional TCP congestion control. It will send at twice the rate limit, causing 50% packet loss to its own traffic and everything else to that same subscriber. Evidently some TCP geniuses have decided to use latency rather than packet loss as the indicator of congestion, and that the objective is goodput not throughput. Works for last mile technologies like T1 and DSL with big buffers at the head end of the fixed speed serial connection, not so good with the type of rate limit queues we tend to use unless we can provision the queues with big buffers. Probably not your problem, but I thought I’d bring it up just in case. From: Kurt Fankhauser Sent: Sunday, April 26, 2015 10:50 AM To: [email protected] Subject: [AFMUG] 450SM sustain bucket throttle not working.. I have a 450 SM that is rate limited in the SM to 1500kbps download on the sustain side. I noticed last night that this customer was pulling a steady almost 3mbps download for several hours on end. How is this possible? Is there a problem with 13.2 firmware? Its a 3.65ghz SM. see attached. Kurt Fankhauser Wavelinc Communications P.O. Box 126 Bucyrus, OH 44820 http://www.wavelinc.com tel. 419-562-6405 fax. 419-617-0110 --
