Re: akamai yesterday - what in the world was that
Yup, Call of Duty update, 68GB on xbox platform. Tom On Tue, Feb 11, 2020 at 10:26 PM Aaron Gould wrote: > Huge! Big as ever. My aanp links are (were) pegged, seriously. I will > be contacting Akamai about lighting up an additional 10 gig link to my > local clusters. Started at 12 noon central… still going pretty heavily. > Game/update release ? > > > > -Aaron > > > > > > *From:* Tom Deligiannis [mailto:tom.deligian...@gmail.com] > *Sent:* Tuesday, February 11, 2020 5:41 PM > *To:* aar...@gvtc.com > *Cc:* Nanog@nanog.org > *Subject:* Re: akamai yesterday - what in the world was that > > > > There is a major update that has released today, how's everything looking > for everyone? > > > > Tom > > > >
RE: akamai yesterday - what in the world was that
Huge! Big as ever. My aanp links are (were) pegged, seriously. I will be contacting Akamai about lighting up an additional 10 gig link to my local clusters. Started at 12 noon central… still going pretty heavily. Game/update release ? -Aaron From: Tom Deligiannis [mailto:tom.deligian...@gmail.com] Sent: Tuesday, February 11, 2020 5:41 PM To: aar...@gvtc.com Cc: Nanog@nanog.org Subject: Re: akamai yesterday - what in the world was that There is a major update that has released today, how's everything looking for everyone? Tom
Re: akamai yesterday - what in the world was that
> Any word on what the update was for? It caused quite a jump in traffic on our > network. On twitter "68 GB" was trending https://twitter.com/search?q=%2268%20GB%22=trend_click Kind regards, Job
Re: akamai yesterday - what in the world was that
Dido On Feb 11, 2020, at 9:03 PM, Andy Smith mailto:telephonetoughgu...@gmail.com>> wrote: Any word on what the update was for? It caused quite a jump in traffic on our network. On Tue, Feb 11, 2020, 19:06 Jared Mauch mailto:ja...@puck.nether.net>> wrote: Looking good from my perspective. Let me know if we are causing you pain and let's see what can be done to improve. I'm here in SF if you are at nanog. Sent from my iCar > On Feb 11, 2020, at 3:42 PM, Tom Deligiannis > mailto:tom.deligian...@gmail.com>> wrote: > > There is a major update that has released today, how's everything looking for > everyone?
Re: akamai yesterday - what in the world was that
Any word on what the update was for? It caused quite a jump in traffic on our network. On Tue, Feb 11, 2020, 19:06 Jared Mauch wrote: > Looking good from my perspective. Let me know if we are causing you pain > and let's see what can be done to improve. > > I'm here in SF if you are at nanog. > > Sent from my iCar > > > On Feb 11, 2020, at 3:42 PM, Tom Deligiannis > wrote: > > > > There is a major update that has released today, how's everything > looking for everyone? >
Re: akamai yesterday - what in the world was that
Looking good from my perspective. Let me know if we are causing you pain and let's see what can be done to improve. I'm here in SF if you are at nanog. Sent from my iCar > On Feb 11, 2020, at 3:42 PM, Tom Deligiannis > wrote: > > There is a major update that has released today, how's everything looking for > everyone?
Re: akamai yesterday - what in the world was that
There is a major update that has released today, how's everything looking for everyone? Tom On Thu, Jan 23, 2020 at 10:14 AM Aaron Gould wrote: > My gosh, what in the word was that coming out of my local Akamai aanp > servers yesterday !? starting at about 12:00 noon central time lasting > several hours ? > > > > -Aaron > On Sat, Jan 25, 2020 at 2:02 PM Tom Deligiannis wrote: > Shouldn't game patches like this be released overnight during off-peak >> hours? Fortnite releases their updates around 3 or 4am when most ISP's >> networks are at their lowest utilization. It seems somewhat reckless to >> release such a large patch during awake hours. >> > > I can't speak for PS4 and PC, but xbox does have a setting to keep the > console in a state that allows the device to download updates when it is > 'off' but I've noticed that the consoles don't always follow that rule and > the update starts downloading when the user powers on the console, which is > usually during peak hours. If this worked properly, it would be great since > most of the updates would download while users were at school, working, > sleeping, etc. > > On Sat, Jan 25, 2020 at 1:36 PM Darin Steffl > wrote: > >> Shouldn't game patches like this be released overnight during off-peak >> hours? Fortnite releases their updates around 3 or 4am when most ISP's >> networks are at their lowest utilization. It seems somewhat reckless to >> release such a large patch during awake hours. >> >> On Sat, Jan 25, 2020, 12:08 PM Brandon Jackson via NANOG >> wrote: >> >>> "Call of Duty: Modern Warfare fragged our business VOIP: US ISP blames >>> outage on smash-hit video game rush >>> This is Windstream, going dark..." >>> https://www.theregister.co.uk/2020/01/23/windstream_fvoip_outage/ >>> >>> Apparently not everyone came out unscathed. >>> >>> -- >>> Brandon Jackson >>> bjack...@napshome.net >>> >>> >>> On Thu, Jan 23, 2020 at 10:14 AM Aaron Gould wrote: >>> My gosh, what in the word was that coming out of my local Akamai aanp servers yesterday !? starting at about 12:00 noon central time lasting several hours ? -Aaron >>>
Re: Customer sending blackhole route with another provider's AS
Chris Adams wrote on 11/02/2020 17:30: > Just curious what others do... I always assumed AS path filtering to > customer (and their downstream customers) AS was a standard best > practice. It is. Then again, there exists every exception to the rule you can think of. If the exception has not been seen yet, we have not looked hard enough. => I.e. it depends. Is my answer. BCP is to not accept direct customer routes 'another provider's AS in the path'. If you can reach an agreement with the customer. You can agree to a >standardized< exception for this single customer. <= Your dice to roll. You are the customers upstream in this case. AS-path rewriting on the customers side of the eBGP connection is an option. If they remove $otherProviders ASN from the path before (re-)announcing the black-hole routes to you. So $customerASN is seen as the source when you receive the announcements. Chriztoffer
Re: Customer sending blackhole route with another provider's AS
Anyone that is using blackhole communities should have enough Clue-fu to adjust announcements along each pathway to have the correct sequence of ASNs. Passing a route with a different upstream's ASN as the origin, instead of their own, is just *asking* for "blackhole leakage", where they inadvertently become a conduit for blackhole prefixes from provider A getting redistributed to you as provider B. Push back on them, and indicate they must pass properly-crafted AS-PATH attributes to you in order to be accepted. If they don't know how to do that, a) they shouldn't be mucking with blackhole communities, and b) they should consider hiring Clue-fu to bring their network policies up to snuff. ^_^; Matt On Tue, Feb 11, 2020 at 8:31 AM Chris Adams wrote: > One of our multihomed customers is set up with some type of security > system from another upstream that can announce blackhole routes for > targeted IPs. They have a BGP policy to take those blackhole routes and > add our blackhole community string so that we can drop the traffic (and > we in turn translate to our transit providers). All good. > > However, it doesn't work, because the route the customer sends to us has > the other upstream's AS as the source, and we have AS path filtering on > our customer links. > > Is this a typical setup? Should we just accept the route(s) with > another provider's AS in the path? That seems... unusual. Our internal > blackhole system uses a private AS (so it can be stripped off before > sending to anyone else). > > Just curious what others do... I always assumed AS path filtering to > customer (and their downstream customers) AS was a standard best > practice. > > -- > Chris Adams > >
Customer sending blackhole route with another provider's AS
One of our multihomed customers is set up with some type of security system from another upstream that can announce blackhole routes for targeted IPs. They have a BGP policy to take those blackhole routes and add our blackhole community string so that we can drop the traffic (and we in turn translate to our transit providers). All good. However, it doesn't work, because the route the customer sends to us has the other upstream's AS as the source, and we have AS path filtering on our customer links. Is this a typical setup? Should we just accept the route(s) with another provider's AS in the path? That seems... unusual. Our internal blackhole system uses a private AS (so it can be stripped off before sending to anyone else). Just curious what others do... I always assumed AS path filtering to customer (and their downstream customers) AS was a standard best practice. -- Chris Adams
Re: CISCO 0-day exploits
Large operators have very little to gain from calling out the equipment suppliers. In my personal experience large operators are already getting custom code builds based on their exact requirements, which include disabling many of the “standard” features they don’t use. Sent from my iPhone > On Feb 11, 2020, at 9:48 AM, Ahmed Borno wrote: > > > Being realistic, as you mentioned, these vendors do not have the right > incentive. > > Thats one thing that operators can do and maybe it should be a recurring > theme at NANOG, calling out vendors to put some sanity and logic into how > iACLs and CoPP are handled. They can do a lot if they cared to spend some $ > on being creative, maybe a BoF for this specific topic. > > Creativity in the form of ways to avoid the fragile stacks and L3 packet of > death, They can even separate the Mgmt plane from the Control plane if they > are serious about it, they can enforce iACL on Mgmt interfaces, they can have > logic to validate packets before they are processed, and to be fair, this > needs to happen in the existing install base too, not only the new ones. I am > trying to say that if they cant hire skilled programmers then they should > show innovation around the most vulnerable part of their code...the trusting > nature of protocols. > > P.S: How many junior network engineers care to turn on authentication on L2 > segments. > > ~A > >> On Tue, Feb 11, 2020 at 6:24 AM Saku Ytti wrote: >> On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: >> >> > Sorry for the sad tone, i just wish network operators would find a way to >> > challenge these vendors and call their less than optimal quality. >> >> It's hard, TINA. We can talk about white label, but in the end of the >> day, that box is just as proprietary as rest of them, because you >> can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs >> were not supported, because vendors thought the specs were their >> secret sauce. >> When some vendor finally releases full specs on github including P4 >> compiler target for their chip and will sell chip on their web for 1 >> unit at x USD, we may start to see some real progress, we can start >> building open source NOS with data-planes. >> >> Maybe INTC could start the revolution with Tofino. Ship PCI cards with >> Tofino and few 100GE ports (local switching support) and open it up >> entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like >> they have lot to lose, considering terrible market performance. >> >> >> -- >> ++ytti
Re: CISCO 0-day exploits
Being realistic, as you mentioned, these vendors do not have the right incentive. Thats one thing that operators can do and maybe it should be a recurring theme at NANOG, calling out vendors to put some sanity and logic into how iACLs and CoPP are handled. They can do a lot if they cared to spend some $ on being creative, maybe a BoF for this specific topic. Creativity in the form of ways to avoid the fragile stacks and L3 packet of death, They can even separate the Mgmt plane from the Control plane if they are serious about it, they can enforce iACL on Mgmt interfaces, they can have logic to validate packets before they are processed, and to be fair, this needs to happen in the existing install base too, not only the new ones. I am trying to say that if they cant hire skilled programmers then they should show innovation around the most vulnerable part of their code...the trusting nature of protocols. P.S: How many junior network engineers care to turn on authentication on L2 segments. ~A On Tue, Feb 11, 2020 at 6:24 AM Saku Ytti wrote: > On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: > > > Sorry for the sad tone, i just wish network operators would find a way > to challenge these vendors and call their less than optimal quality. > > It's hard, TINA. We can talk about white label, but in the end of the > day, that box is just as proprietary as rest of them, because you > can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs > were not supported, because vendors thought the specs were their > secret sauce. > When some vendor finally releases full specs on github including P4 > compiler target for their chip and will sell chip on their web for 1 > unit at x USD, we may start to see some real progress, we can start > building open source NOS with data-planes. > > Maybe INTC could start the revolution with Tofino. Ship PCI cards with > Tofino and few 100GE ports (local switching support) and open it up > entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like > they have lot to lose, considering terrible market performance. > > > -- > ++ytti >
Re: CISCO 0-day exploits
On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: > Sorry for the sad tone, i just wish network operators would find a way to > challenge these vendors and call their less than optimal quality. It's hard, TINA. We can talk about white label, but in the end of the day, that box is just as proprietary as rest of them, because you can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs were not supported, because vendors thought the specs were their secret sauce. When some vendor finally releases full specs on github including P4 compiler target for their chip and will sell chip on their web for 1 unit at x USD, we may start to see some real progress, we can start building open source NOS with data-planes. Maybe INTC could start the revolution with Tofino. Ship PCI cards with Tofino and few 100GE ports (local switching support) and open it up entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like they have lot to lose, considering terrible market performance. -- ++ytti
Re: CISCO 0-day exploits
I remember my conversation with an executive one day, where I was enlightened on corporate greed. I asked, why is there no investment in quality code, and I was schooled. The exec said, one dollar spent on fixing bugs, returns zero dollars but one dollar spent on nee features brings in 3 dollars ;) These vendors tried different DSL throughout the years and it is way too difficult for them to execute on that shift, too many things at stake, i mean their business, not the end customer...of course. They are not even being 'creative', they can easily pull some tricks like the one you mentioned (compiler built sanity) in their system configuration logic, like you cant turn on an interface without an iACL applied...but then why would they :) Sorry for the sad tone, i just wish network operators would find a way to challenge these vendors and call their less than optimal quality. ~A On Tue, Feb 11, 2020 at 2:05 AM Saku Ytti wrote: > On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > > > So yeah iACLs, CoPP and all sorts of basic precautions are needed, but > I'm thinking something more needs to be done, specially if these ancient > code stacks are being imported into new age 'IoT' devices, multiplying the > attack vector by a factor of too many. > > I can't see situation getting better. Why should vendor invest in high > quality code, certainly the cultural shift will cost something, it's > not 0 cost and what is the upside? If IOS and JunOS realistically were > significantly less buggy many of us would stop buying support, because > we either know how to configure these or can get help faster free from > the community, we largely need the support because the software > quality is so bad _everyone_ finds new bugs all the time and we don't > have the source code to fix it as a community. > So I suspect significantly better quality software would at least > initially cost more to produce and it would reduce revenue in loss of > support. > > I also think the way we develop needs to be fundamentally rethought, > we need to stop believing I am the person who can code working C, it's > the other people who are incompetent. At some point we should look, > are the tools we using the right tools? Can we move complexity from > human to computers at compile time to create more guarantees of > correctness? MSFT claims 70% of their bugs are memory safety, issue > which could be solved almost perfectly programmatically by making the > compiler and language smarter, not the person more resistant to > mistakes. > I think ANET at least for some part essentially writes their own DSL > which compiles to C++, I think solution like this for any large > long-lived project probably quickly pays dividends in quality, because > you can address lot of the systematic errors during the compilation > time and in DSL design. > > > -- > ++ytti >
Re: CISCO 0-day exploits
On 2/11/2020 2:04 AM, Saku Ytti wrote: > On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > >> So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm >> thinking something more needs to be done, specially if these ancient code >> stacks are being imported into new age 'IoT' devices, multiplying the attack >> vector by a factor of too many. > > I can't see situation getting better. Why should vendor invest in high > quality code, certainly the cultural shift will cost something, it's > not 0 cost and what is the upside? If IOS and JunOS realistically were > significantly less buggy many of us would stop buying support, because > we either know how to configure these or can get help faster free from > the community, we largely need the support because the software > quality is so bad _everyone_ finds new bugs all the time and we don't > have the source code to fix it as a community. > So I suspect significantly better quality software would at least > initially cost more to produce and it would reduce revenue in loss of > support. Yeah, things need to get better, and soon. At least, some things... Was I too subtle just now? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: CISCO 0-day exploits
On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm > thinking something more needs to be done, specially if these ancient code > stacks are being imported into new age 'IoT' devices, multiplying the attack > vector by a factor of too many. I can't see situation getting better. Why should vendor invest in high quality code, certainly the cultural shift will cost something, it's not 0 cost and what is the upside? If IOS and JunOS realistically were significantly less buggy many of us would stop buying support, because we either know how to configure these or can get help faster free from the community, we largely need the support because the software quality is so bad _everyone_ finds new bugs all the time and we don't have the source code to fix it as a community. So I suspect significantly better quality software would at least initially cost more to produce and it would reduce revenue in loss of support. I also think the way we develop needs to be fundamentally rethought, we need to stop believing I am the person who can code working C, it's the other people who are incompetent. At some point we should look, are the tools we using the right tools? Can we move complexity from human to computers at compile time to create more guarantees of correctness? MSFT claims 70% of their bugs are memory safety, issue which could be solved almost perfectly programmatically by making the compiler and language smarter, not the person more resistant to mistakes. I think ANET at least for some part essentially writes their own DSL which compiles to C++, I think solution like this for any large long-lived project probably quickly pays dividends in quality, because you can address lot of the systematic errors during the compilation time and in DSL design. -- ++ytti
