Didn't you have a bunch of problems with LimeLight in the past?
----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Ken Hohhof" <af...@kwisp.com> To: af@afmug.com Sent: Tuesday, September 20, 2016 1:54:10 PM Subject: Re: [AFMUG] CDN Overload Mike, I know this doesn’t have all the information you are looking for, but it’s all I have time to capture right now. The source IPs seem to be Level3 CDN, and it’s sending just under 6 Mbps of traffic to a customer rate-limited by the tower router to 3 Mbps (Cisco rate limiting which is RED). The torch results are from a Mikrotik router upstream of the tower. The 10 second torch shows around 40 TCP connections. This seems to be a common pattern, push traffic until packet loss is around 50%, with around 50 TCP connections. I tried blocking individual IPs and it was like whack-a-mole, it just added more IPs. Then I blocked 8.0.0.0/8 which did stop the traffic, but I didn’t want to leave that in place. Once I stopped dropping that traffic, it started up again. I don’t know what the traffic is, but I suspect Windows 10 update. It’s a little old lady with one desktop computer. She says it started around 4pm yesterday, which seems a little early for Patch Tuesday. It is making her Internet totally unusable, can’t look up directions, can’t check Facebook, sporadically gets email. From: Mike Hammett Sent: Tuesday, September 20, 2016 9:09 AM To: af@afmug.com Subject: Re: [AFMUG] CDN Overload Can you address the questions I posed in the initial e-mail? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Jim Bouse [Brazos WiFi]" <j...@brazoswifi.com> To: af@afmug.com Sent: Tuesday, September 20, 2016 8:58:12 AM Subject: Re: [AFMUG] CDN Overload I’ve seen it the most from Limelight. Don’t know what they are cramming down my user’s throats but I suspect it is either Microsoft or Apple. Jim Bouse Owner Mobile IT Pro - Brazos WiFi 979-985-5912 j...@brazoswifi.com From: Af [mailto:af-boun...@afmug.com] On Behalf Of Mike Hammett Sent: Monday, September 19, 2016 10:29 PM To: af@afmug.com Subject: Re: [AFMUG] CDN Overload Gather evidence, attempt to work cooperatively, then name and shame if necessary. But yes, that's close to my intention. If you do your homework properly, the greater networking community is very powerful and will back you. Those companies are largely ones that will work with you. Forget Amazon, Sony, etc. though. I've heard from people seeing this with Microsoft, Akamai, Limelight and Apple. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "That One Guy /sarcasm" < thatoneguyst...@gmail.com > To: af@afmug.com Sent: Monday, September 19, 2016 10:16:26 PM Subject: Re: [AFMUG] CDN Overload Did you just indicate an intention to get a cdn to alter a corporate policy? I have a huge satchel, I mean it could probably hold a couple bowling balls, reality only fills it with a couple small pecans. Does it hurt? On Sep 19, 2016 9:43 PM, "Mike Hammett" < af...@ics-il.net > wrote: Have you seen a CDN overloading a customer? Help me gather information on the issue. What CDN? What have you identified the traffic to be? What is the access network? Where is the rate limiting done? How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? What is doing the rate limiting? What is the rate-limit set to? Upstream of the rate-limiter, what are you seeing for inbound traffic? One connection or many? How much traffic? How does other traffic behave when exceeding the rate limit? Where is NAT performed? What is doing NAT? Shared NAT or isolated to that customer? Have you done a packet capture before and after the rate limiter? The NAT device? Would you be willing to send a filtered packet capture (only the frames that relate to this CDN) to the CDN if they want it? There have been reports of CDNs sending more traffic than the customer can handle and ignores TCP convention to slow down. Trying to investigate this thoroughly so we can get the CDN to fix their system. Multiple CDNs have been shown to do this. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
