Re: iOS 7 update traffic
It's back with this: Ben quite succinctly sums it up on a nanog mailing list, “Your (the service provider) user is paying you to push packets. If that’s causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the “cloud” bandwagon, the future is only going to see you pushing more packets, not less ! So if you can’t stand the heat, get out of the kitchen (or the xSP industry).” Thanks Darren http://www.mellowd.co.uk/ccie On 24 Sep 2013, at 23:51, Ben ben+na...@list-subs.com wrote: On 24/09/2013 18:54, Michael Brown wrote: That is most assuredly a rewrite, it's not just your perception. M. Surprise surprise, that page now appears to Error 404... guess he must watch the list quite closely as it didn't take long for him to react ! ;-) Guess I should be flattered someone finds my internet rantings quoteworthy ! Would have appreciated at least a feable attempt at a citation (or at least a generic reference to the Nanog list in general).
Re: iOS 7 update traffic
I have heard a lot of questions and debate about whether the iOS updates download automatically: “Available updates download automatically if your device is connected to Wi-Fi and a power source.” http://support.apple.com/kb/HT4623 http://support.apple.com/kb/HT4623 /mark http://support.apple.com/kb/HT4623 http://support.apple.com/kb/HT4623
Re: iOS 7 update traffic
On Sep 26, 2013, at 5:22 PM, Mark Lancaster markl...@gmail.com wrote: I have heard a lot of questions and debate about whether the iOS updates download automatically: “Available updates download automatically if your device is connected to Wi-Fi and a power source.” http://support.apple.com/kb/HT4623 http://support.apple.com/kb/HT4623 /mark The wording in step 2 is poorly done. The availability of updates is what is downloaded. Step 3 indicates an active user input to begin download is required.
Re: iOS 7 update traffic
Cutler James R james.cut...@consultant.com writes: On Sep 26, 2013, at 5:22 PM, Mark Lancaster markl...@gmail.com wrote: I have heard a lot of questions and debate about whether the iOS updates download automatically: Available updates download automatically if your device is connected to Wi-Fi and a power source. http://support.apple.com/kb/HT4623 http://support.apple.com/kb/HT4623 /mark The wording in step 2 is poorly done. The availability of updates is what is downloaded. Step 3 indicates an active user input to begin download is required. The updates do download automatically, but only if the device is on wifi and power at the same time. For iOS 6, a check for available updates will be attempted at a randomly chosen time on a randomly chosen day of the week. If one is found, an automatic download may follow if/when on power and wifi. Opening the Software Update pane will cause an immediate check for available updates. If one is found it will be displayed to the user, who may touch the button, which will complete the download (even if not on power or not on wifi, although there are minimum battery charge requirements and some updates can't be downloaded over cell) as necessary, and then perform the install. So, an ISP will see initial traffic when an update is released, as people manually install it, and some continuing traffic spread over at least the next week as the automatic downloads occur. Then, of course, once people have updated their device, they'll want to use it: update their apps, download a new Siri voice, buy music...
Re: iOS 7 update traffic
On Mon, Sep 23, 2013 at 08:36:30PM -0500, Joe Greco wrote: That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones. The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images. So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. Sounds like a win to me. - Matt -- Python is a rich scripting language offering a lot of the power of C++ while retaining the ease of use of VBscript. -- The PyWin32 documentation
Re: iOS 7 update traffic
On Tue, 24 Sep 2013, Jay Ashworth wrote: Fixing 4 (which is an easy engineering issue) and 5 (which is an operations policy issue that, by and large, most people in that situation understand), *would have had a direct positive effect on Apple's paying customers*. Fixing 4 is something apple should do. 5 is a marketing decision for them. People are used to queuing *for things*. If the apple updates break your network instead of just the apple updates, then that is your fault as an ISP. For me, nothing more than the apple updates were broken during those hours, that I could tell anyway. The Internet wasn't broken. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On Sep 24, 2013, at 12:45 AM, Jay Ashworth j...@baylink.com wrote: Strawman, Randy. Clearly, the Internet is *not* up to the task of 1) updating several dozen million devices 2) on links of various quality, 3) with 650MB to 1.2GB downloads and 4) a client that doesn't understand how to restart 5) all at once, cause, over all, it went very poorly. It went well for most users, it seems the 1-5% of people with odd configs are the problem. Keep in mind that on any average day about 3% of the networks out there are broken based on pre-ipv6 day measurements. That is, their IPv4 is completely busted. Having the error/problem rate being down in that area is reasonable to me. How many code-red/slammer scans do you still see a decade on? Overall this was surely a network traffic event, and those that observed the IOS6 impact a year ago realized it would occur again with IOS7 and monitored for it. Not everything will work for everyone, but for the majority of users it was fine. (This from surveying my non-geek friends). Traffic levels will lower about 7 days post-release. Also, NYC and other police departments are advocating people update immediately for the anti-theft upgrades provided in the new software. Let me know how your conversation with them goes. - Jared
Re: iOS 7 update traffic
- Original Message - From: Jared Mauch ja...@puck.nether.net It went well for most users, it seems the 1-5% of people with odd configs are the problem. [ ... ] IOS7 and monitored for it. Not everything will work for everyone, but for the majority of users it was fine. (This from surveying my non-geek friends). Hmmm. The number of unsolicited reports I saw and received -- both from upgraders and those who didn't even know it was happening, seemed much higher than would support that. When you're upgrading several hundred million devices, the error rate doesn't have to be that high... But the things you *should* do are still a larger list than the things you *must* do; tragedy of the commons and like that. Apple could *easily* have handled it better. That's my story, and I'm stickin' to it. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
Re: iOS 7 update traffic
Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/ The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. While his argument for installing CDNs is not entirely incorrect, you should take it with a pound of salt, as the author works for a CDN vendor! :-) http://blog.streamingmedia.com/2009/07/alcatellucent-acquires-cdn-technology-provider-velocix.html
Re: iOS 7 update traffic
On 24/09/2013 17:55, Glen Kent wrote: Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/ The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. Hang on a minute. That last paragraph in his blog sounds awfully similar to something I posted here the other day ! He says on the 23rd of September : Users are paying service providers to deliver their IP packets. If providers cant handle the volume of traffic that they’re being asked to deliver then its probably time for them to reevaluate their commercial structure (charge customers more) or to redesign/overhaul their networks (invest in CDNs, etc). Remember, with everyone connecting to the “cloud”, the traffic that providers will be asked to push is only going to increase with time .. I said on the 20th of September : Your user is paying you to push packets. If that's causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the cloud bandwagon, the future is only going to see you pushing more packets, not less ! So if you can't stand the heat, get out of the kitchen (or the xSP industry). Hmm.. ;-(
Re: iOS 7 update traffic
You've been robbed! On 9/24/2013 1:36 PM, Ben wrote: Hang on a minute. That last paragraph in his blog sounds awfully similar to something I posted here the other day ! He says on the 23rd of September : Users are paying service providers to deliver their IP packets. If providers cant handle the volume of traffic that they’re being asked to deliver then its probably time for them to reevaluate their commercial structure (charge customers more) or to redesign/overhaul their networks (invest in CDNs, etc). Remember, with everyone connecting to the “cloud”, the traffic that providers will be asked to push is only going to increase with time .. I said on the 20th of September : Your user is paying you to push packets. If that's causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the cloud bandwagon, the future is only going to see you pushing more packets, not less ! So if you can't stand the heat, get out of the kitchen (or the xSP industry). Hmm.. ;-( -- Jon Sands
Re: iOS 7 update traffic
On 24/09/2013 18:54, Michael Brown wrote: That is most assuredly a rewrite, it's not just your perception. M. Surprise surprise, that page now appears to Error 404... guess he must watch the list quite closely as it didn't take long for him to react ! ;-) Guess I should be flattered someone finds my internet rantings quoteworthy ! Would have appreciated at least a feable attempt at a citation (or at least a generic reference to the Nanog list in general).
Re: iOS 7 update traffic
Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/ The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. http://news.idg.no/cw/art.cfm?id=391B4B64-F693-41B7-6BBAC6D7017C3B8A John From: Colin Alston colin.als...@gmail.com To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: NANOG nanog@nanog.org Sent: Monday, 23 September 2013, 1:05 Subject: Re: iOS 7 update traffic That system by the way is annoying when your mobile network operator are so oversubscribed/old-fashioned that I had to wait over 6 months before I could update to Android ICS... I really don't want my ability to update the software on my phone to be controlled by a teleco, and these large teleco's really should have Akamai caches in place by now - if they even know what that is. On Thu, Sep 19, 2013 at 8:22 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamsson email: swm...@swm.pp.se
Re: iOS 7 update traffic
On 23/09/13 10:32, John Smith wrote: Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/ The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. http://news.idg.no/cw/art.cfm?id=391B4B64-F693-41B7-6BBAC6D7017C3B8A John Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties. The hard bit would be to create a way for Apple etc. to be able to authoritatively say we are the content owners, and are happy for you to replicate this locally: but perhaps this could be as simple serving the initial seed from an HTTPS server with a valid certificate? It would then be trivial to create a whitelist of the domains of the top 10 or so distributors of patches, and then everything would work automatically from then on. -- N.
Re: iOS 7 update traffic
One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct? Do most big service providers maintain their own content servers? Is this what we're heading to these days? Glen On Mon, Sep 23, 2013 at 4:29 PM, Neil Harris n...@tonal.clara.co.uk wrote: On 23/09/13 10:32, John Smith wrote: Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.**com/2013/09/23/ios7s-impact-** on-networks-worldwide/http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/ The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. http://news.idg.no/cw/art.cfm?**id=391B4B64-F693-41B7-**6BBAC6D7017C3B8Ahttp://news.idg.no/cw/art.cfm?id=391B4B64-F693-41B7-6BBAC6D7017C3B8A John Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties. The hard bit would be to create a way for Apple etc. to be able to authoritatively say we are the content owners, and are happy for you to replicate this locally: but perhaps this could be as simple serving the initial seed from an HTTPS server with a valid certificate? It would then be trivial to create a whitelist of the domains of the top 10 or so distributors of patches, and then everything would work automatically from then on. -- N.
Re: iOS 7 update traffic
Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties. They should do it like the game vendors. You are able to preload the game but it will only run on the day it is released. But, you still have to make sure not all your customers fetch the content at the same time with full speed. And what about thouse who just own a phone and no computer with iTunes? These customers will prefer to download over wifi as fast as possible to get the upgrade done. Preloading over anything other than wifi is way to expensive. Someone mentioned Win8/8.1 and Upgrades, there will also be OSX Mavericks quite soon. Fewer devices but way more content ... Access is sold by bandwith, so there will be oversubscrition. Sell it by traffic and all thouse iPhone users will walk to the neares Apple store to buy an USB stick with the upgrade since its cheaper than downloading it. rm
Re: iOS 7 update traffic
Glen Kent writes: One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct? Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers. Do most big service providers maintain their own content servers? Is this what we're heading to these days? Depends on what you mean by their own. As I said, these days Akamai has servers in many of the big networks. Google and possibly others (Limelight, ...?) might have that as well. But I wouldn't call them their [the SPs'] own. Some SPs are also built their own CDNs (Level 3) or are talking about it. But that model seems to be less popular with the content owners and the other SPs. -- Simon.
Re: iOS 7 update traffic
On Sep 23, 2013, at 8:10 AM, Simon Leinen simon.lei...@switch.ch wrote: Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers. Apple claims 200 million[1] IOS devices upgrade to version 7 in the past week. A typical download was on the order of 800MB. At the same time, Apple released some other updates, like OSX 10.8.5[2] (275MB) and XCode 5.0[3] (2GB). They also made the iWork and iLife applications (Pages, Numbers, Keynote iMovie, and iPhoto) free to download[4] for all new IOS purchasers. Oh, and they sold 9 million iPhone 5s/c devices[1], most of which needed an update to IOS 7.0.1[5] which was a 1.2GB download. With all of that going on the grumbling on NANOG has pretty much been limited to yeah, we saw a spike, and a few providers of alternative technologies (like Satellite) grousing a bit. I'm not saying the industry can't do better, but I'm finding it hard to describe what happened as anything besides a success for CDN's and most consumer facing ISP's. I only hope the various CDN's and ISP's study what happened so they can be prepared for the next event, which will no doubt be bigger. We're all in an up and to the right industry. 1: http://9to5mac.com/2013/09/23/apple-announces-9-million-iphone-sales-over-first-three-days/ 2: http://support.apple.com/kb/DL1675 3: http://9to5mac.com/2013/09/18/xcode-5-0-released-with-ios-7-sdk-64-bit-app-compiler/ 4: http://9to5mac.com/2013/09/10/apple-makes-iwork-apps-iphoto-and-imovie-free-with-all-new-ios-devices/ 5: http://support.apple.com/kb/DL1683 -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: iOS 7 update traffic
BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent? Glen On Mon, Sep 23, 2013 at 4:29 PM, Neil Harris n...@tonal.clara.co.uk wrote: On 23/09/13 10:32, John Smith wrote: Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.**com/2013/09/23/ios7s-impact-** on-networks-worldwide/http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/ The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. http://news.idg.no/cw/art.cfm?**id=391B4B64-F693-41B7-**6BBAC6D7017C3B8Ahttp://news.idg.no/cw/art.cfm?id=391B4B64-F693-41B7-6BBAC6D7017C3B8A John Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties. The hard bit would be to create a way for Apple etc. to be able to authoritatively say we are the content owners, and are happy for you to replicate this locally: but perhaps this could be as simple serving the initial seed from an HTTPS server with a valid certificate? It would then be trivial to create a whitelist of the domains of the top 10 or so distributors of patches, and then everything would work automatically from then on. -- N.
Re: iOS 7 update traffic
On 2013-09-23, at 09:10, Simon Leinen simon.lei...@switch.ch wrote: Glen Kent writes: One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct? Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers. I think oversubscription in the access network (between the customer and the ISP network that might contain Akamai nodes) is the general concern, at least from the ISPs I have visibility into. Your access network doesn't have to be a narrowband satellite network for this kind of unexpected demand to hurt, and provisioning extra access bandwidth in anticipation of a one- or two-day possibility of increased demand is not practical. I don't doubt Apple are aware of the issue and will make changes if they can. The characterisation that Apple doesn't care, or is callously causing pain in other networks ignores the commercial reality that user experience is important to them. The user experience when an anticipated update can't be downloaded easily is not great. The suggestions on how to make things better next time that have appeared on this list seem helpful. I would imagine that any vendor with a huge and widely-distributed user base would do well to take note. Joe
Re: iOS 7 update traffic
On Sep 23, 2013, at 15:10, Simon Leinen simon.lei...@switch.ch wrote: Glen Kent writes: One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct? Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers. I have some anecdotal evidence that a large swatch of Telekom land in Germany was fed from two (2) Limelight servers in Frankfurt (?). Of course, packet loss to them during Wednesday evening was around 50 %. (I VPNed out of Telekom land to get my iOS 7 update, which was then no problem at all; that clearly shows that the access infrastructure wasn't overloaded.) It doesn't help that Apple's update software has no way to make use of the results of a prematurely aborted transfer; this is a recipe for bistable behavior. Grüße, Carsten
Re: iOS 7 update traffic
On Sep 23, 2013, at 9:41 AM, Glen Kent glen.k...@gmail.com wrote: BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent? It's more about predictable results and outcome. I can pay a CDN and likely get some sort of reporting/SLA. I can't as easily insure that my torrent traffic will work as well. Some carriers dabbled in doing something about peer-to-peer/torrent type traffic in the past, such as the P4P stuff: http://www.datacenterknowledge.com/archives/2008/03/14/verizon-testing-p4p-for-peer-to-peer-delivery/ But I think it died off like many other things. I think CNN.com video still wants the peer-to-peer octoshape thing, but I have always said NO. https://www.google.com/search?q=octoshape+peer+to+peer while an older article from 2009, here's why you should say no as well: http://arstechnica.com/business/2009/02/cnn-p2p-video-streaming-tech-raises-questions/ - Jared
Re: iOS 7 update traffic
On 2013-09-23 15:41 , Glen Kent wrote: BTW Linux distributions are available to download via bittorrent, I am very sure that you will be happy to see your customer's UPSTREAM links filled with that traffic... next to you having a shiny CDN and then having to do traffic to ISPs who do not have one... IMHO the model with CDN is appropriate for this kind of traffic: the pusher is footing the bill. Greets, Jeroen
Re: iOS 7 update traffic
On 2013-09-23, at 09:41, Glen Kent glen.k...@gmail.com wrote: BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent? There are upstream congestion issues frequently associated with bittorrent. If you compare (a) five thousand students on a campus wifi network trying to download a 1GB image from a nearby Akamai cache, and (b) five thousand students on a campus wifi network seeding a 1GB image to people all over the world it's not obvious that more pain results from (a) than (b). Even given the ability of Apple to control the behaviour of the bittorrent agent (which presumably would be built into iOS) the impact of such a strategy on an event of this size seems very hard to predict, given a narrow time base and an unknowable number of local network constraints. It doesn't seem impossible to try and optimise the fan-out by giving network operators hooks to influence peer selection based on local topology. But it also doesn't sound like an easy general problem to solve (or a problem that anybody necessarily wants to spend money on if the relief is only going to be felt once per year on major iOS updates). (Remember as well that the scale here is very different. With iOS, Apple is the major Unix vendor on the planet by some margin. No other single Linux or other Unix/Unix-like distribution comes close, and I am guessing no single operating system triggers the update enthusiasm observed with iOS.) Joe
Re: iOS 7 update traffic
Bit torrent is a way to lighten the load on the originator, and to increase the speed of the acquisition from the receivers. It is not a tool to decrease network load, if anything it does the opposite most of the time. Every now and then, a client will find a local network peer, but its usually an accident more than anything from the algorithm it uses to try to find the fastest senders with pieces it needs. This is most often a product of far end congestion and what pieces are completed, and rarely upstream related barring major issues. The algorithim is self greedy, not altruistic, and definitely not written with ISP load issues in mind. I'd much prefer CDN content over bittorrent from the ISP side. -Blake On Mon, Sep 23, 2013 at 9:02 AM, Joe Abley jab...@hopcount.ca wrote: On 2013-09-23, at 09:41, Glen Kent glen.k...@gmail.com wrote: BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent? There are upstream congestion issues frequently associated with bittorrent. If you compare (a) five thousand students on a campus wifi network trying to download a 1GB image from a nearby Akamai cache, and (b) five thousand students on a campus wifi network seeding a 1GB image to people all over the world it's not obvious that more pain results from (a) than (b). Even given the ability of Apple to control the behaviour of the bittorrent agent (which presumably would be built into iOS) the impact of such a strategy on an event of this size seems very hard to predict, given a narrow time base and an unknowable number of local network constraints. It doesn't seem impossible to try and optimise the fan-out by giving network operators hooks to influence peer selection based on local topology. But it also doesn't sound like an easy general problem to solve (or a problem that anybody necessarily wants to spend money on if the relief is only going to be felt once per year on major iOS updates). (Remember as well that the scale here is very different. With iOS, Apple is the major Unix vendor on the planet by some margin. No other single Linux or other Unix/Unix-like distribution comes close, and I am guessing no single operating system triggers the update enthusiasm observed with iOS.) Joe
Re: iOS 7 update traffic
- Original Message - From: Mikael Abrahamsson swm...@swm.pp.se To: Paul Ferguson fergdawgs...@mykolab.com On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... It wasn't sinister, Ferg, it was *stupid*. *They announced the release date and time*. Everyone's phone has a check for release now button, and yet they believed that only *push notifying* phones in waves would be enough to prevent what happened. See also: screwed the pooch. This went out, what, last Weds and Thu? I had half a dozen people from all different environments ask me why's the Internet broke today? on release day. There was a consumer-visible impact from this absolutely asinine release engineering project on Apple's part. You don't announce the exact release time, and you don't even *make the release visible to all devices at the same time* since Twitter. This is apparently their first rodeo. We should take pains to make it their last. As an anti-Apple guy, I resent having my stuff screwed up because of it. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
Re: iOS 7 update traffic
That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones. The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images. For the ISP, this would mean minimal amount of effective downloads. On 09/23/2013 07:31 AM, Blake Dunlap wrote: Bit torrent is a way to lighten the load on the originator, and to increase the speed of the acquisition from the receivers. It is not a tool to decrease network load, if anything it does the opposite most of the time. Every now and then, a client will find a local network peer, but its usually an accident more than anything from the algorithm it uses to try to find the fastest senders with pieces it needs. This is most often a product of far end congestion and what pieces are completed, and rarely upstream related barring major issues. The algorithim is self greedy, not altruistic, and definitely not written with ISP load issues in mind. I'd much prefer CDN content over bittorrent from the ISP side.
Re: iOS 7 update traffic
That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones. The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images. So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. And then - assuming you intend for more things than just Apple to go this route - all the CDN's would need to be redesigned to support BT too. It seems like it'd be simpler for Apple to figure out how to validate a partial download and then resume. It isn't like that would be cutting edge technology. I think I might even have seen it happen before. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: iOS 7 update traffic
On 9/23/2013 9:36 PM, Joe Greco wrote: So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. Just ask the Blizzard fans (World of Warcraft) about this phenomenon... Jeff
Re: iOS 7 update traffic
So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. Just ask the Blizzard fans (World of Warcraft) about this phenomenon... i love the business plan of preventing the users from getting what they want. i think all my competitors should follow it. randy, who uses a bunch of bittorrent, try btsync for more private dropbox
Re: iOS 7 update traffic
- Original Message - From: Randy Bush ra...@psg.com i love the business plan of preventing the users from getting what they want. i think all my competitors should follow it. Strawman, Randy. Clearly, the Internet is *not* up to the task of 1) updating several dozen million devices 2) on links of various quality, 3) with 650MB to 1.2GB downloads and 4) a client that doesn't understand how to restart 5) all at once, cause, over all, it went very poorly. The people negatively impacted by that poor engineering planning on Apple's part *are Apple's customers*, quite apart from any negative impact it had on Everyone Else. Fixing 4 (which is an easy engineering issue) and 5 (which is an operations policy issue that, by and large, most people in that situation understand), *would have had a direct positive effect on Apple's paying customers*. Preventing them from getting what they want is made up, and I'm pretty sure you know that. Staging FW update rollouts over networks is old hat. Even I know better than to make that mistake, and I don't know anything. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
Re: iOS 7 update traffic
On 09/23/2013 08:36 PM, Joe Greco wrote: That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones. The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images. So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. No, usually that traffic is demoted right before upstream (or in some way not very near to the provider-edge-to-customer device). Once the download is ready on the ISP, that would be a solved problem. And also, the phone could support two protocols as a transition. It's the easiest solution I've read so far. There are others but not as easy. And then - assuming you intend for more things than just Apple to go this route - all the CDN's would need to be redesigned to support BT too. Why can't it be implemented as an independent mean of delivering updates? It seems like it'd be simpler for Apple to figure out how to validate a partial download and then resume. It isn't like that would be cutting edge technology. I think I might even have seen it happen before. Validate partial download? How would that help to reduce the overall load on the ISP? That is only limited to reducing the redundant traffic, where redundant means twice per device, not twice per content, which is the real problem. Cheers.
Re: iOS 7 update traffic
That system by the way is annoying when your mobile network operator are so oversubscribed/old-fashioned that I had to wait over 6 months before I could update to Android ICS... I really don't want my ability to update the software on my phone to be controlled by a teleco, and these large teleco's really should have Akamai caches in place by now - if they even know what that is. On Thu, Sep 19, 2013 at 8:22 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
http://www.electronista.com/articles/13/09/20/upgrading.spike.doubled.some.isp.traffic.12.percent.worldwide.internet.usage.jump/ ## Upgrading spike doubled some ISP traffic; 12 percent worldwide Internet usage jump ... Analytics company Mixpanel estimates that more than 130 million users had updated their devices within the first 10 hours of availability, out of a potential iOS 7-eligible base of 415 million. ... The spikes, reported by The Guardian in the UK, focus on British ISP reports and show that demand hit its peak about 3.5 hours after the official release. This would have been around the time Apple servers began to recover from the initial wave of downloaders as users struggled in the first few hours to download the free iOS upgrade, which ranged in size from 750MB to 1.4GB depending on what device was being updated. ... At one point in the initial surge, a British Telecom (BT) spokesperson reported that the provider was seeing traffic of over 200 Gigabits per second, the highest the company has ever recorded. Lonap said that overall high-use traffic, which averages up to 30 Gigabits per second, spiked to just under 60Gb per second late Wednesday evening as the upgrade became available, and raised the daily peak time traffic to nearly 40Gb per second on Thursday. The paper also reported that Germany's Berlin Internet Exchange saw traffic rise precipitously -- more than 10Gb per second over normal traffic levels -- as iOS 7 came online. Data from Akamai, a company that distributes and manages Internet backbone traffic, suggested that overall worldwide usage jumped 12 percent above normal levels as the download became available. ##
Re: iOS 7 update traffic
Just as a note. On Sep 19, 2013, at 6:57 PM, Brandon Galbraith wrote: 1) Rate limit the software update download (Us) 2) Have device OS download the update in the background, and be resilient to failures with retries (Manufacturer) Apple already does this in the iTunes update the ios device mode. 3) Don't present the update notification to the user until the update blob is already cached on the device (Manufacturer) Apple also already does this. However, manual checks/updates can be done. When there is so much buzz on the news and given Apple customers zeal a large percentage manually invoke the update. Only in a perfect world though. On Thu, Sep 19, 2013 at 5:49 PM, joel jaeggli joe...@bogus.com wrote: On 9/19/13 3:29 PM, Warren Bailey wrote: Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking.. Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity? Who else has had traffic surges like this? Flash traffic occurs, sometimes people fly planes into things, sometimes nuclear reactors melt down, earthquakes or hurricanes occur or cables are segmented due to underwater landslides. and what infrastructure that is left shifts abruptly from terrestrial to sattelite or gets droppped on the floor. the best you can ask for on an instantanious basis is graceful degredation under load. this happens to not be weather.so maybe you can do something about it. but ultimately a certain number of bytes have to be transfered and given the architecture, the flash was driven by the consumer and not by software automation, if we want the later to control it consumer choice has to be taken out of the loop, which may or may not be palatable. And who else has a Nanog strike team coming in screaming buy more bandwidth? ;) Sent from my Mobile Device. Original message From: Ryan Harden harde...@uchicago.edu Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart jer...@mompl.net Cc: nanog@nanog.org nanog@nanog.org Subject: Re: iOS 7 update traffic On Sep 19, 2013, at 3:11 PM, Jeroen van Aart jer...@mompl.net wrote: On 09/19/2013 12:06 PM, Ryan Harden wrote: As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. The difference is there are now a couple more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you. The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing. This isn't a new phenomenon. I realize some of you are too cool for Apple Lame low ball remark, however I thought it was the opposite, Apple==coolness? This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's iJunk. I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out. Regards, Jeroen -- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
Re: iOS 7 update traffic
On 19/09/2013 9:29 PM, Jeff Kell wrote: On 9/19/2013 5:29 PM, Warren Bailey wrote: So you understand things aren't always metro e.. That's what I was trying to say. I still have a coupler.. ;) Original message From: Fred Reimer frei...@freimer.org Actually, I started out with a 300 baud acoustic modem. You know, the kind where you take the handset and jam it into two cups? But I digress… Bah! That was a take-home convenience. How about the old ASR TeleType with the 110-baud link to get a hardcopy listing? Jeff Ah, distinctly I remember. The ASR shook the whole house when it started typing things out. I was using a government tie line from Ottawa to Toronto, and occasionally the operator would break in to investigate the long call with funny noises. Tom
Re: iOS 7 update traffic
* Ben (ben+na...@list-subs.com) wrote: No need for you to bash Apple in this instance. What this conversation badly needs is a sub-thread about whatever happened to the technical solutions which would address this issue (eg: mbone). Of course, I know what happened and what the issues are there, but it'd be fun to watch people complain about how Apple should try and do the right thing and make multicast work for their updates, 'cause it's the right technical solution. Thanks, Stephen signature.asc Description: Digital signature
Re: iOS 7 update traffic
Why does apple feel it is okay to send every mobile device an update on a single day? (a) That's why god invented the concept of CDNsto take the stress of the more contended parts of an operators network. ;-) (b) Its not just Apple but any vendor (e.g. Microsloth) their updates are released to the world at the same time. (c) Your user is paying you to push packets. If that's causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the cloud bandwagon, the future is only going to see you pushing more packets, not less ! So if you can't stand the heat, get out of the kitchen (or the xSP industry). ;-) No need for you to bash Apple in this instance.
Re: iOS 7 update traffic
On Sep 19, 2013, at 7:13 PM, Mark Andrews ma...@isc.org wrote: Oh you mean that option that never made it past a internet-draft that expired 13 years ago[1] and is in the private range[2] to boot. If you want proxy discovery to work on all devices complete the process of getting a code point allocated then get the OS vendors to query for it. 252 is fine for experimenting / proof of concept but it really is the wrong value for long term use. Mark [1] http://tools.ietf.org/html/draft-ietf-wrec-wpad-01 [2] http://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters.xhtml Sure! I've found that Microsoft devices honor this option, but others do not. I would be in support of something similar to provide this support, but the part of my original reply you missed is that the content is deliberately not-cachable on the part of either the CDN or the originator. Microsoft patches are also not easily cacheable as well because they only request about 100Kb per request, so you get an awful lot of HTTP/206. They also make it easier to run local caches for an enterprise. The apple process requires the full patch to come down in one-shot and doesn't like being interrupted. It might be easier for Warren to ship each customer a 16GB USB with the whole set of images for each device type. Then again, they would have to know how to use them I know what to do, but my other family members, not so much... - Jared
Re: iOS 7 update traffic
In message f11ff3cf-d363-4af0-a030-b72cd68dd...@puck.nether.net, Jared Mauch writes: On Sep 19, 2013, at 7:13 PM, Mark Andrews ma...@isc.org wrote: Oh you mean that option that never made it past a internet-draft that expired 13 years ago[1] and is in the private range[2] to boot. If you want proxy discovery to work on all devices complete the process of getting a code point allocated then get the OS vendors to query for it. 252 is fine for experimenting / proof of concept but it really is the wrong value for long term use. Mark [1] http://tools.ietf.org/html/draft-ietf-wrec-wpad-01 [2] http://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameter s.xhtml Sure! I've found that Microsoft devices honor this option, but others do not. I would be in support of something similar to provide this support, but the part of my original reply you missed is that the content is deliberately not-cachable on the part of either the CDN or the originator. Microsoft patches are also not easily cacheable as well because they only request about 100Kb per request, so you get an awful lot of HTTP/206. They also make it easier to run local caches for an enterprise. The apple process requires the full patch to come down in one-shot and doesn't like being interrupted. It might be easier for Warren to ship each customer a 16GB USB with the whole set of images for each device type. Then again, they would have to know how to use them I know what to do, but my other family members, not so much... - Jared So you fix one part at a time. Each part is independently fixable. Each part helps by itself. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: iOS 7 update traffic
On 9/20/2013 5:55 PM, Mark Andrews wrote: So you fix one part at a time. Each part is independently fixable. Each part helps by itself. I wonder it the thing that needs to be fixed first involves opening a dialog between Engineering and Marketing over which the message Don't sell stuff we can't deliver might be passed. And just maybe some we are going public with a new whatever in 6 months, here is [some meaningful amount of money], get us ready. messages in the opposite direction? -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)
Re: iOS 7 update traffic
From a Facebook posting: So need the masses input.. If you updated to iOS 7... Wed I installed it, it was fine. Thursday fine as well. But today.. It just wants to keep resetting itself every 15-20 mins. It's just on my 4s... Any one else having this issue? Just wondering. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)
Re: iOS 7 update traffic
I've seen 4s with the panic.list problem after upgrade. Apple claims hardware issue. On Fri, Sep 20, 2013 at 8:13 PM, Larry Sheldon larryshel...@cox.net wrote: From a Facebook posting: So need the masses input.. If you updated to iOS 7... Wed I installed it, it was fine. Thursday fine as well. But today.. It just wants to keep resetting itself every 15-20 mins. It's just on my 4s... Any one else having this issue? Just wondering. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)
Re: iOS 7 update traffic
Haven't updated my iPad yet but the iPhone update size was 1.12GB On Sep 19, 2013, at 2:05 PM, Mikael Abrahamsson wrote: On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On 9/19/13 10:58 AM, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... iOS 7 itself was implemented. ~Seth
Re: iOS 7 update traffic
IOS7 was released (Yesterday?). Due to the large number of IOS devices out in the world. Some network operators experienced large spikes in traffic as each device was updated (Downloading the update). You see the same thing happen when new software is released from people like Microsoft. Or, If you're a gamer. When a new game is released on a digitally delivered platform like Steam or EA's Origin. Nick Olsen Network Operations (855) FLSPEED x106 From: Paul Ferguson fer...@people.ops-trust.net Sent: Thursday, September 19, 2013 1:58 PM To: n...@flhsi.com, nanog@nanog.org Subject: Re: iOS 7 update traffic Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... I think this was just the traffic to download iOS 7 to everyones' relevant Apple devices. I don't know how large the update was (maybe a few hundred MB per device?), but I guess everyone got the notification or their devices started automatically downloading around the same time. The vast majority of the traffic here (large .edu) happened between about 1 and 5 PM yesterday. jms
Re: iOS 7 update traffic
Major update provides many of 5S functionality to the 5, 4S, 4 On Sep 19, 2013, at 1:58 PM, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
RE: iOS 7 update traffic
It was released Thanks Darren http://www.mellowd.co.uk/ccie Date: Thu, 19 Sep 2013 10:58:24 -0700 From: fergdawgs...@mykolab.com To: nanog@nanog.org Subject: Re: iOS 7 update traffic Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon
Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On 2013-09-19, at 13:58, Paul Ferguson fergdawgs...@mykolab.com wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... I think the inference is that iOS 7 caused the extra traffic by being available for download. There are just a lot of Apple devices, and they tend to get upgraded more promptly than other platforms (e.g. on release day). We saw a similar phenomenon tracking downloads of the root zone DNSSEC trust anchor from data.iana.org -- we now see three million downloads per month, and pretty much all of those are iOS devices (or other devices impersonating them, which seems unlikely). Joe
Re: iOS 7 update traffic
Okay, that makes sense. Just wanted to ensure it wasn't something more sinister. Thanks, - ferg On 9/19/2013 11:05 AM, Mikael Abrahamsson wrote: On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
Tens of millions of devices multiplied times a fairly large download = lots of bandwidth. It has an appreciable affect on the worldwide Internet. I would love to see some aggregate statistics. With most phones the carrier takes care of doing phone software updates and rollouts over a period of time since they all have customized versions of Android/Windows Phone/etc. Apple controls their phone software so they just hit the switch at a certain data/time and let as many people update as their servers can handle. Not to mention all the IPads which they chose to update at the same time. Last time around we saw a sustained increase in traffic for about a week after the release date. -Phil On 9/19/13 1:58 PM, Paul Ferguson fergdawgs...@mykolab.com wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
Composed on a virtual keyboard, please forgive typos. On Sep 19, 2013, at 13:58, Paul Ferguson fergdawgs...@mykolab.com wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... BING for ios adoption rate (one estimate is 29% in 16 hours), multiply by # of iThings, multiply by size of iOS, divide by # of seconds in estimate. As for why so many users upgrade so fast, that's a harder question. It could be iThing users are more willing to believe the fruit company's advertising (hype) . Could be that the device tells them to upgrade so they do. It is also at least partially due to the fact all iThings are upgradable (within a certain age horizon). Hope that gives you something to chew on, even if it doesn't answer the question. -- TTFN, patrick On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
http://images.mirror.co.uk/upl/m4/jun2011/6/0/image-5-for-riots-break-out-a fter-vancouver-canucks-lose-the-nhl-stanley-cup-playoffs-to-the-boston-brui ns-gallery-116084753.jpg Good example of the flash crowds post hockey championship It's not all butterflies, Abley.. LOL On 9/19/13 11:42 AM, Joe Abley jab...@hopcount.ca wrote: On 2013-09-19, at 14:11, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? How is this different from the flash crowds caused by hockey championships, or football games, or any of the other things that generate a lot of simultaneous interest every once in a while? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Given that the code is signed, I'm surprised that iDevices that have already upgraded the hard way don't advertise a update available service on local networks. Individual devices don't care where the updates come from, so long as the signatures are good. You'd think that'd have the potential to improve the user experience as well as avoid jamming the tubes, especially in highly multi-user environments like university campuses; it could probably halve the network load in a significant number of home networks, too. Joe
Re: iOS 7 update traffic
A line, is a line, is a line, is a line. There's no difference. Updates are available to all devices on a download day, and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, ATT wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard. On 9/19/13 11:43 AM, Cutler James R james.cut...@consultant.com wrote: On Sep 19, 2013, at 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: Why does apple feel it is okay to send every mobile device an update on a single day? Apple does not send updates. The user device must request an update. --As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade. James R. Cutler james.cut...@consultant.com
Re: iOS 7 update traffic
Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time. On 9/19/13 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
Composed on a virtual keyboard, please forgive typos. On Sep 19, 2013, at 14:11, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? That question makes no sense to me. Turn that around: Why would Apple think that is not OK? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Most providers are offered a cache for free (there is a minimum traffic volume, but it is not even as large as Netflix's requirements). Every provider, regardless of traffic, is offered peering for free. What was the problem again? -- TTFN, patrick Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On Thu, 19 Sep 2013 18:11:11 -, Warren Bailey said: Why does apple feel it is okay to send every mobile device an update on a single day? How is Patch Tuesday any different? pgpOIAByDzs9N.pgp Description: PGP signature
Re: iOS 7 update traffic
How about add a cache in your time capsule/thing that everyone connects to? I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's. We own and operate a satellite data network. It is narrow band.. It is very expensive. Oversub matters.. Time slots matter. When we (industry, collectively) have to deal with hundreds of gigs worth of traffic it's a tough day. I just can't understand why these big boys have forgotten most of the planet doesn't have a 100gbps pipe anywhere near them, a lot of the earth actually doesn't have communications infrastructure at all. The quick WE HAVE ENOUGH INTERNETS NOW doesn't apply to every system, nor does it explain why an update needs to be sent relentlessly to individual devices (requested or not) over the course of a product's evolution. Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc. We strive to provide a great customer experience, and when Hardware Maker X decides to roll updates .. It can screw us. In this case, can means absolutely will happen. My .02. Not trying to start a flame thing or tell people what's what.. Just trying to get a point across. You don't need to send things individually now.. We live in the future.. We should act like it. On 9/19/13 12:10 PM, Fred Reimer frei...@freimer.org wrote: O.K., I understand. Yes, for the average user I suppose they would blame their ISP. I was making the wrong assumption that people understood how the Internet works. At the same time, people would probably be more upset, at least the Apple fanboys, if they metered the updates and some people had to wait two or three weeks for their update to keep the traffic manageable. The only general news stories I see in a quick search are complaints that the downloads are slow, not that the general Internet is slow because of the downloadsŠ From: Warren Bailey wbai...@satelliteintelligencegroup.com Reply-To: Warren Bailey wbai...@satelliteintelligencegroup.com Date: Thursday, September 19, 2013 2:52 PM To: Fred Reimer frei...@freimer.org, Mikael Abrahamsson swm...@swm.pp.se, Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic My.. Our.. Users expect one thing.. Internet. It is our job to make that happen. When a electronics manufacturer decides to enable updates for all of their phones world wide.. It breaks things. When the Internet breaks, it is my fault. Your Apple update sucked because of me.. There is no it must be apple, as you pointed out earlier. I'm simply saying.. It's a dick move to globally enable updates on a single day and tell ISP's to deal with it. Sent from my Mobile Device. Original message From: Fred Reimer frei...@freimer.org Date: 09/19/2013 11:48 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com,Mikael Abrahamsson swm...@swm.pp.se,Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time. On 9/19/13 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times
Re: iOS 7 update traffic
(merging 2 replies) On Thu, 19 Sep 2013 19:11:21 -, Warren Bailey said: Patch Tuesday is not 1gb per patch. It is those months a service pack comes out. On Thu, 19 Sep 2013 18:22:50 -, Warren Bailey said: These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. No - their actions *are* those of smart guys. They knew damned well that they didn't need to add capacity, because the chances that they'll lose an iOS customer to Android over slow update speeds is infinitesimal. In other words, they successfully made the bandwidth consumption an externality. (What, like you don't look at *your* network and wonder How can I stick somebody else with the cost of XYZ?. :) pgpQMCvJse05L.pgp Description: PGP signature
Re: iOS 7 update traffic
On 13-09-19 02:46 PM, Warren Bailey wrote: A line, is a line, is a line, is a line. There's no difference. Updates are available to all devices on a download day, and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, ATT wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard. You fail at internet, please try again later.
Re: iOS 7 update traffic
Patch Tuesday is not 1gb per patch. On 9/19/13 11:51 AM, valdis.kletni...@vt.edu valdis.kletni...@vt.edu wrote: On Thu, 19 Sep 2013 18:11:11 -, Warren Bailey said: Why does apple feel it is okay to send every mobile device an update on a single day? How is Patch Tuesday any different?
Re: iOS 7 update traffic
On 2013-09-19, at 14:11, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? How is this different from the flash crowds caused by hockey championships, or football games, or any of the other things that generate a lot of simultaneous interest every once in a while? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Given that the code is signed, I'm surprised that iDevices that have already upgraded the hard way don't advertise a update available service on local networks. Individual devices don't care where the updates come from, so long as the signatures are good. You'd think that'd have the potential to improve the user experience as well as avoid jamming the tubes, especially in highly multi-user environments like university campuses; it could probably halve the network load in a significant number of home networks, too. Joe
Re: iOS 7 update traffic
On Sep 19, 2013, at 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: Why does apple feel it is okay to send every mobile device an update on a single day? Apple does not send updates. The user device must request an update. --As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade. James R. Cutler james.cut...@consultant.com
Re: iOS 7 update traffic
Woah there. I think you are crossing another line, or at least opening another topic of discussion, when you start talking about transit or last mile providers charging companies for bandwidth that their customers are already paying for. I'd suggest a subject change if we want to open a discussion on that topic. On 9/19/13 2:46 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: A line, is a line, is a line, is a line. There's no difference. Updates are available to all devices on a download day, and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, ATT wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard. On 9/19/13 11:43 AM, Cutler James R james.cut...@consultant.com wrote: On Sep 19, 2013, at 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: Why does apple feel it is okay to send every mobile device an update on a single day? Apple does not send updates. The user device must request an update. --As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade. James R. Cutler james.cut...@consultant.com
Re: iOS 7 update traffic
Sounds like a great plan. You could do it for Netflix, Hulu, amazon, Walmart, etc. Get a piece of the action.Am I talking to Verizon? On 9/19/13 1:46 PM, Warren Bailey wrote: A line, is a line, is a line, is a line. There's no difference. Updates are available to all devices on a download day, and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, ATT wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard.
Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Cutler James R wrote: --As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade. That's supposed to be the nature of upgrades. If that's where the matter ended then you'd have no argument. The problem is when it comes to the new bugs that get introduced, and whether they affect features you care about. I'm holding back until next week :) (And after previous bitter experience with over-the-air upgrade, I've downloaded it through iTunes to a desktop and will upgrade it by wire at my leisure). Jethro. . . . . . . . . . . . . . . . . . . . . . . . . . Jethro R Binks, Network Manager, Information Services Directorate, University Of Strathclyde, Glasgow, UK The University of Strathclyde is a charitable body, registered in Scotland, number SC015263.
Re: iOS 7 update traffic
On Sep 19, 2013, at 3:10 PM, Fred Reimer frei...@freimer.org wrote: I was making the wrong assumption that people understood how the Internet works. Absolutely! Most people understand that the internet works by use of a browser and are content with that knowledge. Much like most motor vehicle operators understand accelerator, brake, steering, and gas pump, with no knowledge of thermodynamics, much less physics or traffic laws. James R. Cutler james.cut...@consultant.com
Re: iOS 7 update traffic
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued. Microsoft and others have their own strategies for incremental downloads, caching, etc.. Apple has theirs. Seems like most consumers want the update and are actively fetching it vs having older software live forever and not be updated. Overall I see this as a win. Jared Mauch On Sep 19, 2013, at 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day?
Re: iOS 7 update traffic
Absolutely correct. Large file updates etc are not an issue for wide band communications networks. If you have 10mbps to your house with a 30ms delay to your first hop.. You're sitting pretty. If you have a 1mbps/512kbps pipe with a built in 750ms latency, things get a little more complicated. Add in the fact that our bandwidth is insanely expensive (link budgets put 1mbps are at about 1MHz *1bits/hz* which costs roughly $3800 a month in sky access ONLY). We have some challenges, and we try (sometimes actually succeed) to get data from point A to point B in an acceptable time frame. The issue is.. When a user begins to EXCEED their general usage things in contention land get pretty hairy. If my 500 remote locations are running on 10MHz worth of capacity (about 10mbps down, 2mbps up) at a 10:1 or 20:1 oversub things get tight sometimes. Imagine an entire user base trying to update their electronic devices on something that feels like 56k. All things on the internet are not created equal, and I can accept that. I can accept that I/we have challenges, but I would figure that those of you who had easier lives would want to share that ease with us who are doing things a little differently. I'm not saying poor us, we need love.. I'm just saying.. I would really appreciate if those of you in the industry who are responsible for these types of scenarios could think about the people who are paying a lot of money, for a very little amount of bandwidth. We don't tier our service by usage cap, but if Apple/Microsoft/whoever increases their update rate we may have to end up looking at something like that. Big files, over a little pipe, with little margin for improvement, is hard. It wouldn't be *THAT* hard to help out those who are trying to make communications available to a much larger customer base than a traditional ISP. Group Hug. //warren On 9/19/13 1:08 PM, Fred Reimer frei...@freimer.org wrote: I certainly don't want to put words in his mouth, but I thin Warren's problem is that he can't upgrade his pipes. Physics limits the bandwidth available, as I think he is a satellite provider. My argument is that if I'm a satellite user I should be well aware, particularly because this is not a new phenomenon, that there are times when my bandwidth will suck. It is what it is. On 9/19/13 3:06 PM, Ryan Harden harde...@uchicago.edu wrote: To be honest, I don't see this as a problem at all. Use it as an excuse to upgrade your pipes, talk Akamai or CDN of choice into putting a cache on your network, or implement your own caching solution. As operators of the Internet we should be looking for ways to enable things like this, not be up in arms at Apple for releasing an update to their phone OS or making it available in a way that's inconvenient to our oversubscription policies. As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. This isn't a new phenomenon. I realize some of you are too cool for Apple, but paying attention to traffic trends and keeping abreast of how new software releases might affect your utilization is part of properly running a network. /Ryan Ryan Harden Senior Network Engineer University of Chicago - AS160 P: 773-834-5441 On Sep 19, 2013, at 1:22 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamssonemail: swm
Re: iOS 7 update traffic
My iPhone4 was about 600MB IIRC. My iPad mini was about that. I have about 7 iDevices between everyone in my immediate family. FWIW not a single one has actually received the notification yet. I've only manually done my 2 devices. I'm waiting to see how long it takes before I get the 'official' notification of an update on the others. On Thu, Sep 19, 2013 at 11:12 AM, TR Shaw ts...@oitc.com wrote: Haven't updated my iPad yet but the iPhone update size was 1.12GB On Sep 19, 2013, at 2:05 PM, Mikael Abrahamsson wrote: On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
+1 If you do not/cannot have an Akamai cache, connect to an IX that does, and make sure you've got the capacity. My own rule of thumb is have 2x the capacity of your average *peak* traffic on an IX. When big events happen, whether it is news, sporting or a major software update, that extra capacity will be sorely needed. At TorIX, most peers traffic jumped by the same percentage that others have bandied about on this thread. One peer jumped almost 100%, but they had the right port speed and thus no issues (at least on the Exchange). Compared to transit in Canada, IX peering is dirt cheap, and pays dividends. -- Stephen On 19/09/2013 3:07 PM, Jared Mauch wrote: The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued. Microsoft and others have their own strategies for incremental downloads, caching, etc.. Apple has theirs. Seems like most consumers want the update and are actively fetching it vs having older software live forever and not be updated. Overall I see this as a win. Jared Mauch On Sep 19, 2013, at 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day?
Re: iOS 7 update traffic
On 9/18/2013 1:38 PM, Zachary McGibbon wrote: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon Traffic was +1Gbps more than usual during that time on 9/18
Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : The vast majority of the traffic I saw was served from the Akamai farm at an upstream provider, so the pain that was felt 'on the backbone' was mitigated somewhat by that. jms
Re: iOS 7 update traffic
Apple actually tries to rate-limit the notifications to prevent this, but you can just manually go check and hit the upgrade button yourself. It's pretty well-known that Apple likes to release ~10am, so tens (hundreds?) of millions of users did just that. Since this update is available for all iThingies made in the last 4-ish years that means a lot of extra traffic. On Thu, Sep 19, 2013 at 7:13 AM, Justin M. Streiner strei...@cluebyfour.org wrote: On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... I think this was just the traffic to download iOS 7 to everyones' relevant Apple devices. I don't know how large the update was (maybe a few hundred MB per device?), but I guess everyone got the notification or their devices started automatically downloading around the same time. The vast majority of the traffic here (large .edu) happened between about 1 and 5 PM yesterday. jms
RE: iOS 7 update traffic
Apple pushed out a new software upgrade for their user interface...a pretty big upgrade, all the iphone users are downloading it congesting the network. Garrison Carr This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. NTT America makes no warranty that this email is error or virus free. Thank you . -Original Message- From: Paul Ferguson [mailto:fergdawgs...@mykolab.com] Sent: Thursday, September 19, 2013 10:58 AM To: NANOG Subject: Re: iOS 7 update traffic Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
On Thu, Sep 19, 2013 at 06:11:11PM +, Warren Bailey wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? They didn't make it available to everyone on the same day. My iphone 5 didn't see it until today; I looked yesterday, and it wasn't available for that device. I'm busily contributing to the network stress now. -- Mike Andrews, W5EGO mi...@mikea.ath.cx Tired old sysadmin
Re: iOS 7 update traffic
They implemented fanboy-lust which :-) Paul Ferguson fergdawgs...@mykolab.com wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote: We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106 From: Justin M. Streiner strei...@cluebyfour.org Sent: Wednesday, September 18, 2013 6:19 PM To: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote: We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too. I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday. jms Zachary McGibbon wrote on 18/9/2013 20:38: So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike? [image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7] Zachary McGibbon -- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID -- Connect and Collaborate -- www.internetidentity.com
Re: iOS 7 update traffic
I might agree if there were no warning, but this has happened a few times a year for many years. It's a predictable pattern and well known. It will last about a week and taper off. Jared Mauch On Sep 19, 2013, at 2:52 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: It's a dick move to globally enable updates on a single day and tell ISP's to deal with it.
Re: iOS 7 update traffic
To be honest, I don't see this as a problem at all. Use it as an excuse to upgrade your pipes, talk Akamai or CDN of choice into putting a cache on your network, or implement your own caching solution. As operators of the Internet we should be looking for ways to enable things like this, not be up in arms at Apple for releasing an update to their phone OS or making it available in a way that's inconvenient to our oversubscription policies. As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. This isn't a new phenomenon. I realize some of you are too cool for Apple, but paying attention to traffic trends and keeping abreast of how new software releases might affect your utilization is part of properly running a network. /Ryan Ryan Harden Senior Network Engineer University of Chicago - AS160 P: 773-834-5441 On Sep 19, 2013, at 1:22 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamssonemail: swm...@swm.pp.se signature.asc Description: Message signed with OpenPGP using GPGMail
Re: iOS 7 update traffic
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On Thu, Sep 19, 2013 at 02:42:12PM -0400, Joe Abley wrote: Given that the code is signed, I'm surprised that iDevices that have already upgraded the hard way don't advertise a update available service on local networks. Individual devices don't care where the updates come from, so long as the signatures are good. Going the other way, Apple will have local update caching as part of MDM for iOS 7 when it is fully upgraded and rolled out to support iOS for enterprises. But the big push with BYOD is that employees manage their own iDevices..
Re: iOS 7 update traffic
My.. Our.. Users expect one thing.. Internet. It is our job to make that happen. When a electronics manufacturer decides to enable updates for all of their phones world wide.. It breaks things. When the Internet breaks, it is my fault. Your Apple update sucked because of me.. There is no it must be apple, as you pointed out earlier. I'm simply saying.. It's a dick move to globally enable updates on a single day and tell ISP's to deal with it. Sent from my Mobile Device. Original message From: Fred Reimer frei...@freimer.org Date: 09/19/2013 11:48 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com,Mikael Abrahamsson swm...@swm.pp.se,Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time. On 9/19/13 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On Thu, Sep 19, 2013 at 06:52:51PM +, Warren Bailey wrote: My.. Our.. Users expect one thing.. Internet. Isn't the ability to download something that they want part of the Internet thing that users expect from their service providers? -dorian
Re: iOS 7 update traffic
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... all the borders and highlights from the discarded skeuomorphisms cloged up the intertubes bigtime randy
Re: iOS 7 update traffic
I certainly don't want to put words in his mouth, but I thin Warren's problem is that he can't upgrade his pipes. Physics limits the bandwidth available, as I think he is a satellite provider. My argument is that if I'm a satellite user I should be well aware, particularly because this is not a new phenomenon, that there are times when my bandwidth will suck. It is what it is. On 9/19/13 3:06 PM, Ryan Harden harde...@uchicago.edu wrote: To be honest, I don't see this as a problem at all. Use it as an excuse to upgrade your pipes, talk Akamai or CDN of choice into putting a cache on your network, or implement your own caching solution. As operators of the Internet we should be looking for ways to enable things like this, not be up in arms at Apple for releasing an update to their phone OS or making it available in a way that's inconvenient to our oversubscription policies. As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. This isn't a new phenomenon. I realize some of you are too cool for Apple, but paying attention to traffic trends and keeping abreast of how new software releases might affect your utilization is part of properly running a network. /Ryan Ryan Harden Senior Network Engineer University of Chicago - AS160 P: 773-834-5441 On Sep 19, 2013, at 1:22 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote: Why does apple feel it is okay to send every mobile device an update on a single day? They don't, these are users who actively goes into the software upgrade menu and pressing upgrade. I believe the nagging won't start for quite some time. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
On 09/19/2013 12:06 PM, Ryan Harden wrote: As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. The difference is there are now a couple more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you. This isn't a new phenomenon. I realize some of you are too cool for Apple Lame low ball remark, however I thought it was the opposite, Apple==coolness? Regards, Jeroen -- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? As far as I was aware, it was at least staggered throughout the day, so there's some concession. Also a reason to have some CDNs in any large deployment, I guess. I saw a spike in our Akamai traffic, but only slight. Sent from my Mobile Device.
Re: iOS 7 update traffic
O.K., I understand. Yes, for the average user I suppose they would blame their ISP. I was making the wrong assumption that people understood how the Internet works. At the same time, people would probably be more upset, at least the Apple fanboys, if they metered the updates and some people had to wait two or three weeks for their update to keep the traffic manageable. The only general news stories I see in a quick search are complaints that the downloads are slow, not that the general Internet is slow because of the downloadsŠ From: Warren Bailey wbai...@satelliteintelligencegroup.com Reply-To: Warren Bailey wbai...@satelliteintelligencegroup.com Date: Thursday, September 19, 2013 2:52 PM To: Fred Reimer frei...@freimer.org, Mikael Abrahamsson swm...@swm.pp.se, Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic My.. Our.. Users expect one thing.. Internet. It is our job to make that happen. When a electronics manufacturer decides to enable updates for all of their phones world wide.. It breaks things. When the Internet breaks, it is my fault. Your Apple update sucked because of me.. There is no it must be apple, as you pointed out earlier. I'm simply saying.. It's a dick move to globally enable updates on a single day and tell ISP's to deal with it. Sent from my Mobile Device. Original message From: Fred Reimer frei...@freimer.org Date: 09/19/2013 11:48 AM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com,Mikael Abrahamsson swm...@swm.pp.se,Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time. On 9/19/13 2:11 PM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Sent from my Mobile Device. Original message From: Mikael Abrahamsson swm...@swm.pp.se Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson fergdawgs...@mykolab.com Cc: NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Paul Ferguson wrote: Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: iOS 7 update traffic
Microsoft Windows 8.1 is due out in October.. don't be so sure :) -- Stephen On 19/09/2013 3:11 PM, Warren Bailey wrote: Patch Tuesday is not 1gb per patch. On 9/19/13 11:51 AM, valdis.kletni...@vt.edu valdis.kletni...@vt.edu wrote: On Thu, 19 Sep 2013 18:11:11 -, Warren Bailey said: Why does apple feel it is okay to send every mobile device an update on a single day? How is Patch Tuesday any different?
Re: iOS 7 update traffic
On Thu, 19 Sep 2013 19:18:29 -, Warren Bailey said: Reversing a few paragraphs to make a point. We strive to provide a great customer experience, and when Hardware Maker X decides to roll updates .. It can screw us. In this case, can means absolutely will happen. I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's So why didn't you? Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc. And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be. pgp__OjVdX58w.pgp Description: PGP signature
Re: iOS 7 update traffic
I'm willing and open to hear anyone who has successfully had that conversation with their users. When network congestion occurs, we typically see a mass exodus from whatever website was being used to Speedtest.. You know.. Just to make sure the internets are fast. I'm trying to highlight a point that not all of us have studly 1gbps connections to Akamai. Some of us have to move data into orbit and back.. Some of us are not like the rest of you. These types of situations should not happen in general.. We live in the future. This is like sending a bulk fax to every user on a switch, and when the other users get busy signals I somehow need to realign my view of reality. A single Internet point or software update should not cause all of this discussion. You guys are collectively posting hundreds of gbps for basically a single software update, and comparing it to point releases from vendors. Why do I feel like many of you are spoiled with all of this cheap and fast bandwidth? Do you guys not remember your 9600bps modem? Many of you would have suffered heart failure if I sent you a 100mb file only 10 years ago. Keep that in mind.. Not everyone has their Internet coming off the end of an sfp. Sent from my Mobile Device. Original message From: valdis.kletni...@vt.edu Date: 09/19/2013 1:42 PM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.com Cc: Fred Reimer frei...@freimer.org,Mikael Abrahamsson swm...@swm.pp.se,Paul Ferguson fergdawgs...@mykolab.com,NANOG nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013 19:18:29 -, Warren Bailey said: Reversing a few paragraphs to make a point. We strive to provide a great customer experience, and when Hardware Maker X decides to roll updates .. It can screw us. In this case, can means absolutely will happen. I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's So why didn't you? Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc. And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be.
RE: iOS 7 update traffic
Hi Jared. The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued. Because it impacts other, non-Apple customers. Or, it costs the ISP more (passed through to all customers) to add capacity to handle an infrequent peak load. Question/suggestion: Could Apple perhaps shift their release to a Saturday morning? I would think that this would go a long way to diluting the peak. John John Souvestre - New Orleans LA - (504) 454-0899 smime.p7s Description: S/MIME cryptographic signature
Re: iOS 7 update traffic
Dorian, It seems warren may work for a satellite internet provider. (Just guessing). The impact might be different with this type of a link. There isn't a good broadcast caching system for this compared with the way other content is. This may have that type of an impact, but there are likely ways to address this as well. I recall the sky cache and other systems of the day. Jared Mauch On Sep 19, 2013, at 2:41 PM, Dorian Kim dor...@blackrose.org wrote: On Thu, Sep 19, 2013 at 06:52:51PM +, Warren Bailey wrote: My.. Our.. Users expect one thing.. Internet. Isn't the ability to download something that they want part of the Internet thing that users expect from their service providers? -dorian
Re: iOS 7 update traffic
Actually, I started out with a 300 baud acoustic modem. You know, the kind where you take the handset and jam it into two cups? But I digress… From: Warren Bailey wbai...@satelliteintelligencegroup.commailto:wbai...@satelliteintelligencegroup.com Reply-To: Warren Bailey wbai...@satelliteintelligencegroup.commailto:wbai...@satelliteintelligencegroup.com Date: Thursday, September 19, 2013 5:00 PM To: Valdis Kletnieks valdis.kletni...@vt.edumailto:valdis.kletni...@vt.edu Cc: Fred Reimer frei...@freimer.orgmailto:frei...@freimer.org, Mikael Abrahamsson swm...@swm.pp.semailto:swm...@swm.pp.se, Paul Ferguson fergdawgs...@mykolab.commailto:fergdawgs...@mykolab.com, NANOG nanog@nanog.orgmailto:nanog@nanog.org Subject: Re: iOS 7 update traffic I'm willing and open to hear anyone who has successfully had that conversation with their users. When network congestion occurs, we typically see a mass exodus from whatever website was being used to Speedtest.. You know.. Just to make sure the internets are fast. I'm trying to highlight a point that not all of us have studly 1gbps connections to Akamai. Some of us have to move data into orbit and back.. Some of us are not like the rest of you. These types of situations should not happen in general.. We live in the future. This is like sending a bulk fax to every user on a switch, and when the other users get busy signals I somehow need to realign my view of reality. A single Internet point or software update should not cause all of this discussion. You guys are collectively posting hundreds of gbps for basically a single software update, and comparing it to point releases from vendors. Why do I feel like many of you are spoiled with all of this cheap and fast bandwidth? Do you guys not remember your 9600bps modem? Many of you would have suffered heart failure if I sent you a 100mb file only 10 years ago. Keep that in mind.. Not everyone has their Internet coming off the end of an sfp. Sent from my Mobile Device. Original message From: valdis.kletni...@vt.edumailto:valdis.kletni...@vt.edu Date: 09/19/2013 1:42 PM (GMT-08:00) To: Warren Bailey wbai...@satelliteintelligencegroup.commailto:wbai...@satelliteintelligencegroup.com Cc: Fred Reimer frei...@freimer.orgmailto:frei...@freimer.org,Mikael Abrahamsson swm...@swm.pp.semailto:swm...@swm.pp.se,Paul Ferguson fergdawgs...@mykolab.commailto:fergdawgs...@mykolab.com,NANOG nanog@nanog.orgmailto:nanog@nanog.org Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013 19:18:29 -, Warren Bailey said: Reversing a few paragraphs to make a point. We strive to provide a great customer experience, and when Hardware Maker X decides to roll updates .. It can screw us. In this case, can means absolutely will happen. I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's So why didn't you? Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc. And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be.
Re: iOS 7 update traffic
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking.. Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity? Who else has had traffic surges like this? And who else has a Nanog strike team coming in screaming buy more bandwidth? ;) Sent from my Mobile Device. Original message From: Ryan Harden harde...@uchicago.edu Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart jer...@mompl.net Cc: nanog@nanog.org nanog@nanog.org Subject: Re: iOS 7 update traffic On Sep 19, 2013, at 3:11 PM, Jeroen van Aart jer...@mompl.net wrote: On 09/19/2013 12:06 PM, Ryan Harden wrote: As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. The difference is there are now a couple more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you. The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing. This isn't a new phenomenon. I realize some of you are too cool for Apple Lame low ball remark, however I thought it was the opposite, Apple==coolness? This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's iJunk. I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out. Regards, Jeroen -- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
