Re: COVID-19 vs. our Networks
Its already happening in Italy, and now that schools are shutting down here as well, its going to get interesting: https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-strain-network-with-fortnite-marathon The ultimate traffic test is coming, looking forward to hearing about it on this thread. Maybe its a good time to start a communication channel between content providers/gaming companies and ISPs/CDNs. On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl wrote: > > > On Thu, Mar 12, 2020 at 3:46 PM g...@1337.io wrote: > >> With talk of there being an involuntary statewide (WA) and then national >> quarantines (house arrest) for multiple weeks, has anyone put thought into >> the impacts of this on your networks if/when this comes to fruition? >> >> We're already pushing the limits with telecommuters / those that are WFH, >> but I can only imagine what things will look like with everyone stuck at >> home for any duration of time. >> > > > People will turn to you and every other ISP hoping you keep them online. > So besides demand issues, keeping your network up will be important to a > whole lot of people. > > > Rubens > >
Re: akamai yesterday - what in the world was that
Strictly out of interest, I wanted to ask earlier if this irresponsible way of causing insane, instant, bandwidth demands is breaking anything on the ISP/CDN side or even the console owner ?! Or is it just an interesting phenomenon that is handled without a sweat. Does it break the buck in anyway? The thread started with bandwidth surges and now power hogging is mentioned, I wonder what else might happen as a side effect to a small number of console/gaming companies not taking a direct responsibility in how they release large updates in a way that is not organized or scheduled but is rough and abrupt. ~A On Thu, Feb 13, 2020 at 3:33 AM Tom Beecher wrote: > The discussion about what the consoles can or can not do is honestly not > solving anything. > > Saying that the consoles should or should not be doing a thing is simply > trying to throw the problem to someone else. > > On Wed, Feb 12, 2020 at 15:40 Carsten Bormann wrote: > >> On 2020-02-12, at 20:45, Mike Hammett wrote: >> > >> > Aren't most modern consoles on whether they're "on" or not? IE: It's >> not a full power up from a dead stop, 0 watts power usage. >> >> >> https://www.anandtech.com/show/7528/the-xbox-one-mini-review-hardware-analysis/5 >> says two-digit standby power (which they say is needed for background >> updating). At least in Germany, nobody sane will leave the thing in that >> expensive mode (a watt-year is $3 here). Switchable extension power cords >> are being actively marketed here for these power hogs. >> >> Grüße, Carsten >> >>
Re: CISCO 0-day exploits
Being realistic, as you mentioned, these vendors do not have the right incentive. Thats one thing that operators can do and maybe it should be a recurring theme at NANOG, calling out vendors to put some sanity and logic into how iACLs and CoPP are handled. They can do a lot if they cared to spend some $ on being creative, maybe a BoF for this specific topic. Creativity in the form of ways to avoid the fragile stacks and L3 packet of death, They can even separate the Mgmt plane from the Control plane if they are serious about it, they can enforce iACL on Mgmt interfaces, they can have logic to validate packets before they are processed, and to be fair, this needs to happen in the existing install base too, not only the new ones. I am trying to say that if they cant hire skilled programmers then they should show innovation around the most vulnerable part of their code...the trusting nature of protocols. P.S: How many junior network engineers care to turn on authentication on L2 segments. ~A On Tue, Feb 11, 2020 at 6:24 AM Saku Ytti wrote: > On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: > > > Sorry for the sad tone, i just wish network operators would find a way > to challenge these vendors and call their less than optimal quality. > > It's hard, TINA. We can talk about white label, but in the end of the > day, that box is just as proprietary as rest of them, because you > can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs > were not supported, because vendors thought the specs were their > secret sauce. > When some vendor finally releases full specs on github including P4 > compiler target for their chip and will sell chip on their web for 1 > unit at x USD, we may start to see some real progress, we can start > building open source NOS with data-planes. > > Maybe INTC could start the revolution with Tofino. Ship PCI cards with > Tofino and few 100GE ports (local switching support) and open it up > entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like > they have lot to lose, considering terrible market performance. > > > -- > ++ytti >
Re: CISCO 0-day exploits
I remember my conversation with an executive one day, where I was enlightened on corporate greed. I asked, why is there no investment in quality code, and I was schooled. The exec said, one dollar spent on fixing bugs, returns zero dollars but one dollar spent on nee features brings in 3 dollars ;) These vendors tried different DSL throughout the years and it is way too difficult for them to execute on that shift, too many things at stake, i mean their business, not the end customer...of course. They are not even being 'creative', they can easily pull some tricks like the one you mentioned (compiler built sanity) in their system configuration logic, like you cant turn on an interface without an iACL applied...but then why would they :) Sorry for the sad tone, i just wish network operators would find a way to challenge these vendors and call their less than optimal quality. ~A On Tue, Feb 11, 2020 at 2:05 AM Saku Ytti wrote: > On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > > > So yeah iACLs, CoPP and all sorts of basic precautions are needed, but > I'm thinking something more needs to be done, specially if these ancient > code stacks are being imported into new age 'IoT' devices, multiplying the > attack vector by a factor of too many. > > I can't see situation getting better. Why should vendor invest in high > quality code, certainly the cultural shift will cost something, it's > not 0 cost and what is the upside? If IOS and JunOS realistically were > significantly less buggy many of us would stop buying support, because > we either know how to configure these or can get help faster free from > the community, we largely need the support because the software > quality is so bad _everyone_ finds new bugs all the time and we don't > have the source code to fix it as a community. > So I suspect significantly better quality software would at least > initially cost more to produce and it would reduce revenue in loss of > support. > > I also think the way we develop needs to be fundamentally rethought, > we need to stop believing I am the person who can code working C, it's > the other people who are incompetent. At some point we should look, > are the tools we using the right tools? Can we move complexity from > human to computers at compile time to create more guarantees of > correctness? MSFT claims 70% of their bugs are memory safety, issue > which could be solved almost perfectly programmatically by making the > compiler and language smarter, not the person more resistant to > mistakes. > I think ANET at least for some part essentially writes their own DSL > which compiles to C++, I think solution like this for any large > long-lived project probably quickly pays dividends in quality, because > you can address lot of the systematic errors during the compilation > time and in DSL design. > > > -- > ++ytti >
Re: CISCO 0-day exploits
Disclaimer, I do not work for any vendor right now, and I don't sell any product that might benefit from scaring anyone, so this is just some whining for a real issue that someone needs to do something about. I've worked for the CDP vendor for a long time, and I do concur to what Saku is saying...the L3 packet of death [threat] is very real, and I'd like to take this opportunity (the buzz around CDPwn) to say a thing or two about these 'soft, mushy and vulnerable' code stacks we have running all over the world, under our feet, waiting for someone with the right incentive to take advantage of. IMHO, "Skilled" software developers, and in parallel...'software exploiters/reverse engineers' haven't been paying attention to these 'infrastructure' boxes (for now), maybe it is because they always had other pieces of the technology stack to work with, and these other tech. stacks were much more rewarding to spend time on (I'm quite sure Node.js / Kubernetes for example...will have a lot more vulnerability researchers looking at them than CDP/LLDP/SNMPetc code). < and That is a serious sustainability issue on our hands, the risk here is very high; when it comes to infrastructure security of nations, especially in a world where miscreants are no longer script kiddies but actual nation sponsored soldiers...Even MBS is doing it in person. Because the moment some miscreants from some oppressive regime decides to do damage, and not necessarily remote code execution as many might think, but more on the 'L3 packet of death' kind of situation that Saku mentioned earlier, these miscreants have a lot to play with, and the attacks vector is huge, it is green and it is ready for the ripe. In my life time, I've looked at so many 'DDTS' descriptions, and I saw nothing but an unwritten disclaimer of : 'I can be easily used for DDoS'...and that is the case even if *SIRT did their brief analysis of these bugs, so again, if some miscreants found it in themselves to look at bugs with the right 'optics', we are going to be in an interesting situation. Luckily, we haven't seen a CDPwn/STPwn/BGPwn/NTPwn/*.*Pwn...etc worm/ransomware yet, but we also don't have reason to think its not possible, and to make matters worse, the code these babies are running is ancient (in every possible way), many of the libraries used to develop that code is glibc_ishy like in nature, and to make matters a bit more interesting, patching those babies is not easy, and the nature of their software architecture makes them even much more fragile than any piece of cheap IP camera out there on the internet or on enterprise networks. So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm thinking something more needs to be done, specially if these ancient code stacks are being imported into new age 'IoT' devices, multiplying the attack vector by a factor of too many. ~A On Mon, Feb 10, 2020 at 5:42 AM Saku Ytti wrote: > On Mon, 10 Feb 2020 at 13:52, Jean | ddostest.me via NANOG > wrote: > > > I really thought that more Cisco devices were deployed among NANOG. > > > > I guess that these devices are not used anymore or maybe that I > > understood wrong the severity of this CVE. > > Network devices are incredibly fragile and mostly work because no one > is motivated to bring the infrastructure down. Getting any arbitrary > vendor down if you have access to it on L2 is usually so easy you > accidentally find ways to do it. > There are various L3 packet of deaths where existing infra can be > crashed with single packet, almost everyone has no or ridiculously > broken iACL and control-plane protection, yet business does not seem > to suffer from it. > > Probably lower availability if you do upgrade your devices just > because there is a known issue, due to new production affecting > issues. > > -- > ++ytti >
Re: DiViNetworks
How is it technically possible that they reuse unused bandwidth without some funky AS/Route announcement fun?! Anyone can explain that ? ~A On Thu, Feb 6, 2020 at 8:09 PM Ronald F. Guilmette wrote: > I mention in passing also that at the present time, DiViNetworks has > a grand total of some 6,070 unique route objects registered in the RADB > data base. > > Where I come from, that's a lot of routes. > >https://pastebin.com/raw/YeFBd1qZ > > I would be gnerally unconcerned if not for the fact that two of these > route objects (for 155.235.0.0/16 and 169.129.0.0/16) exactly cover > two AFRINIC legacy blocks that I feel I have proven to have been stolen > from AFRINIC legacy blocks holders, with the apparent collusion and > connivance of one particular gentleman who, coincidentally, I'm sure, > like DiViNetworks, also just happens to have offices in the greater > Tel Aviv metropolitan area. > > > Regards, > rfg > > > P.S. Online reports suggest that DiViNetworks has received $15 million > USD worth of venture capital from the International Finance Corporation, > a commercial lender and member of the World Bank Group. > > > https://ifcext.ifc.org/ifcext/pressroom/IFCPressRoom.nsf/0/52F1A9E272AAFAB785257BE80051CB53 > > https://en.wikipedia.org/wiki/International_Finance_Corporation >
Re: abrupt speed changes and TCP
I am only guessing here, but I think the Apps of today would have their own built in mechanisms to work around lower layers, starting with DB query timeouts, load balancers performance based resets. CDN segmentation, QUIC, HTTP2etc But it is a valid question and I'd like to know from people with real experience in TCP performance impact of 4 to 5G switching. ~A On Thu, Jan 30, 2020 at 10:59 AM Michael Thomas wrote: > > So it occurs to me in the rollout of 5G just walking down the street you > might shift back and forth between high speed 5G bands and 4G because of > uneven deployment and all sorts of other reasons. It sounds like this > could vary block by block practically. > > I assume TCP just views this as congestion? But with all of the > congestion avoidance algorithms and the rapidly fluctuating bandwidth, > wouldn't that result in the sender essentially adapting to the least > common denominator (eg 4G)? The same goes with latency, I suppose for > real time apps. > > Mike > >
Re: akamai yesterday - what in the world was that
Bezos phone sending Videos to MBS :) What a S Show. On Thu, Jan 23, 2020 at 7:22 AM Jared Mauch wrote: > > > > > On Jan 23, 2020, at 10:16 AM, Kaiser, Erich wrote: > > > > Yeah we saw that as well. Must be a game release or something. > > Yes, that’s my understanding as well. > > - Jared
