FCC Seeks Comment on Effects of June 15 T-Mobile Outage
FCC Public Safety and Homeland Security Bureau Seeks Comment on Effects of June 15, 2020 T-Mobile Outage on Public Safety Entities, Government Entities, and Consumers https://www.fcc.gov/document/fcc-seeks-comment-effects-june-15-t-mobile-outage From T-Mobile's statement: https://www.t-mobile.com/news/update-for-customers-on-network-issues "The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls." Other notable national telecommunication failures AT - January 15 1990, AT Long Distance Network Collapse MCI Worldcom - August 1999, Multi-day frame-relay network failure Centurylink - December 27, 2018, 37-hour nationwide outage
Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?)
> On Jun 23, 2020, at 4:16 AM, Masataka Ohta > wrote: > > Mark Tinka wrote: > >>> But, it should be noted that a single class B... >> CIDR - let's not teach the kids old news :-). > > Saying /16 is ambiguous depends on IP version. Not really… A /16 in IPv6 is a lot more addresses, but it’s still using the first 16 bits to specify the prefix, same as IPv4. Owen
Fiber in the power space
We are looking for an engineering firm with significant experience in FTTX in the power space. Extra points if you have worked with Co-ops. -- Fletcher Kittredge GWI 207-602-1134 www.gwi.net
Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?)
On Mon, Jun 22, 2020 at 7:18 PM Randy Bush wrote: > how did that work out for the ptts? :) > Though its release slipped by three years, by 1995 ATM had started to replace IP as the protocol of choice. By 1999, IP was used only by a small number of academic networks. Nah, I don't think there is anywhere in the multiverse where fat pipes and dumb switches doesn't win. -- Fletcher Kittredge GWI 207-602-1134 www.gwi.net
BGP flooding
> > > So to sum it up you simply can not run into any scaling ceiling with > MP-BGP > > architecture. > > Flooding nature of BGP requires all the related entities treat > everything, regardless of whether they need it entirely or not. That is long gone I am afraid ... Hint RFC 4684. Now applicable to more and more AFI/SAFIs. Also from day one of L3VPNs, PEs even if receiving all routes were dropping on inbound (cheap operation) those routes which contained no locally intersecting RTs. Thx, R.
Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?)
adamv0...@netconsultings.com wrote: The key takeaway however is that no single entity in SP network, be it PE, or RR, or ASBR, ever needs everything, you can always slice and dice indefinitely. So to sum it up you simply can not run into any scaling ceiling with MP-BGP architecture. Flooding nature of BGP requires all the related entities treat everything, regardless of whether they need it entirely or not. Masataka Ohta
Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?)
Masataka Ohta wrote: The point of Yakov on day one was that, flow driven approach of Ipsilon does not scale and is unacceptable. Though I agree with Yakov here, we must also eliminate all the flow driven approaches by MPLS or whatever. I still don't see them in practice, even though they may have been proposed. I don't know, either, as it's Adam who said: > But MPLS can be made flow driven (it can be made whatever the > policy dictates), for instance DSCP driven… Masataka Ohta
Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?)
Mark Tinka wrote: Personally, the level of intelligence we have in routers now beyond being just Layer 1, 2, 3 - and maybe 4 - crunching machines is just as far as I'm willing to go. Once upon a time in Japan, NTT proudly announced to have developed and actually deployed telephone exchangers to be able to offer complex calculator service including trigonometric/exponential/logarithmic functions, which was impossible by handheld calculators at that time. My favorite example when I explain the E2E principle. Masataka Ohta
Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?)
Mark Tinka wrote: But, it should be noted that a single class B... CIDR - let's not teach the kids old news :-). Saying /16 is ambiguous depends on IP version. And, if I understand BGP-MP correctly, all the routing information of all the customers is flooded by BGP-MP in the ISP. Yes, best practice is in iBGP. Some operators may still be using an IGP for this. It would work, but scales poorly. The amount of flooded traffic is not so different. Then, it should be a lot better to let customer edges encapsulate L2 or L3 over IP, with which, routing information within customers is exchanged by customer provided VPN without requiring extra overhead of maintaining customer local routing information by the ISP. You mean like IP-in-IP or GRE? That already happens today, without any intervention from the ISP. I know, though I didn't know ISP's are not offering SLA for it. There are few ISP's who would be able to terminate an IP or GRE tunnel on-net, end-to-end. And even then, they might be reluctant to offer any SLA's because those tunnels are built on the CPE, typically outside of their control. The condition to offer SLA beyond a network of an ISP should not "trusted NNI" but policing by the ISP with ISP's own equipment, which prevent too much traffic enter the network. If ISP's didn't make money from MPLS/VPN's, router vendors would not be as keen on adding the capability in their boxes. It is like telco was making money by expensive telephone exchangers only to be replaced by ISPs, I'm afraid. Label stacking is fundamental to the "MP" part of MPLS. Whether your payload is IP, ATM, Ethernet, Frame Relay, PPP, HDLC, e.t.c., the ability to stack labels is what makes an MPLS network payload agnostic. There is value in that. What? You are saying "payload" not something carrying "payload" is MP. Then, plain Ethernet is MP with EtherType, isn't it? Masataka Ohta
Re: Is there any data on packet duplication?
On Mon, Jun 22, 2020 at 5:30 PM Hal Murray wrote: > > > How often do packets magically get duplicated within the network so that the > target receives 2 copies? That seems like something somebody at NANOG might > have studied and given a talk on. > > Any suggestions for other places to look? bugs like https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn71311 where both hw forwarded and punted packet are sent to destination?
Re: 60 ms cross-continent
Many of the traders have set up their short wave radio transmitters for use across the Atlantic. Bandwidth is only 4 kliobits, but that is enough to send a message saying "buy the SPY Option contracts". It is quite a bit faster than fiber. Regards, Roderick. From: NANOG on behalf of adamv0...@netconsultings.com Sent: Tuesday, June 23, 2020 10:20 AM To: nanog@nanog.org ; l...@satchell.net Subject: RE: 60 ms cross-continent > Stephen Satchell via NANOG > Sent: Monday, June 22, 2020 8:37 PM > > On 6/22/20 12:59 AM, adamv0...@netconsultings.com wrote: > >> William Herrin > >> > >> Howdy, > >> > >> Why is latency between the east and west coasts so bad? Speed of > >> light accounts for about 15ms each direction for a 30ms round trip. > >> Where does the other 30ms come from and why haven't we gotten rid of > it? > >> > > Wallstreet did :) > > https://www.wired.com/2012/08/ff_wallstreet_trading/ > > “Of course, you’d need a particle accelerator to make it work.” > > So THAT'S why CERN wants to build an even bigger accelerator than the LHC! > Yep, why to go around the planet chasing a perfect geodesic with as few relay towers or drones if you can go through (shortest distance is always a straight line as opposed to an arc). While maintaining the speed of light in vacuum since neutrinos don't seem interact with regular matter, that's why they are so darn hard to detect. All you need is an extremely powerful neutrino detector to get you above the 51:49 success ratio. (49% packet loss is not what we're accustomed to, but for these guys it's low enough to start making money). It's quite a fascinating networking world these guys live in, working for a HFT company would be my dream job, always pushing the envelope, racing to the bottom, it's like F1 of the networking world just without the safety and fairness BS to slow you down. adam
RE: 60 ms cross-continent
> Stephen Satchell via NANOG > Sent: Monday, June 22, 2020 8:37 PM > > On 6/22/20 12:59 AM, adamv0...@netconsultings.com wrote: > >> William Herrin > >> > >> Howdy, > >> > >> Why is latency between the east and west coasts so bad? Speed of > >> light accounts for about 15ms each direction for a 30ms round trip. > >> Where does the other 30ms come from and why haven't we gotten rid of > it? > >> > > Wallstreet did :) > > https://www.wired.com/2012/08/ff_wallstreet_trading/ > > “Of course, you’d need a particle accelerator to make it work.” > > So THAT'S why CERN wants to build an even bigger accelerator than the LHC! > Yep, why to go around the planet chasing a perfect geodesic with as few relay towers or drones if you can go through (shortest distance is always a straight line as opposed to an arc). While maintaining the speed of light in vacuum since neutrinos don't seem interact with regular matter, that's why they are so darn hard to detect. All you need is an extremely powerful neutrino detector to get you above the 51:49 success ratio. (49% packet loss is not what we're accustomed to, but for these guys it's low enough to start making money). It's quite a fascinating networking world these guys live in, working for a HFT company would be my dream job, always pushing the envelope, racing to the bottom, it's like F1 of the networking world just without the safety and fairness BS to slow you down. adam
Re: Is there any data on packet duplication?
b...@herrin.us said: > NTP you say? How does iburst work during initial sync up? How does it work, or how should it work? 1/2 :) NTP has been around for a long time. It looks very simple, so anybody thinks they can toss off an implementation without much thought. It will probably work, mostly. The response from an NTP server includes a timestamp that the client put into the request. The client can use that to reject delayed responses to a previous request. When I first started looking for duplicates, I found lots of them. They were NTP version 1 requests. NTP is up to version 4. Version 1 came out in 1988, RFC 1059. Since the requests are identical, there is no way for the client to separate expected responses from delayed responses from a previous request. Does anybody happen to know what equipment or software or OS/distro is sending version 1 requests? -- These are my opinions. I hate spam.
Re: Is there any data on packet duplication?
Am Montag, 22. Juni 2020, 23:53:44 schrieb William Herrin: > On Mon, Jun 22, 2020 at 10:21 PM Saku Ytti wrote: > > On Tue, 23 Jun 2020 at 08:12, William Herrin wrote: > > > That's what spanning tree and its compatriots are for. Otherwise, > > > ordinary broadcast traffic (like those arp packets) would travel in a > > > loop, flooding the network and it would just about instantly collapse > > > when you first turned it on. > > > > Metro: S1-S2-S3-S1 > > PE1: S1 > > PE2: S2 > > Customer: S3 > > STP blocking: ANY > > > > S3 sends frame, it is unknown unicast flooded, S1+S2 both get it > > (regardless of which metro port blocks), which will send it via PE to > > Internet. > > There's a link in the chain you haven't explained. The packet which > entered at S3 has a unicast destination MAC address. That's what was > in the arp table. If they're following the standards, only one of PE1 > and PE2 will accept packets with that destination mac address. The > other, recognizing that the packet is not addressed to it, drops it. > > Recall that ethernet worked without duplicating packets back in the > days of hubs when all stations received all packets. This is how. > > > That having been said, I've seen vendors creatively breach the > boundary between L2 and L3 with some really peculiar results. AWS VPCs > for example. But then this ring configuration doesn't exist in an AWS > VPC and I've not particularly observed a lot of packet duplication out > of Amazon. > > Regards, > Bill Herrin They don't have to break anything or get creative , just assume vrrp between the PE Routers. Not sure how many vendors drop by default if they are not the active router. Regards Karsten
Re: Is there any data on packet duplication?
On Tue, 23 Jun 2020 at 09:54, William Herrin wrote: > There's a link in the chain you haven't explained. The packet which > entered at S3 has a unicast destination MAC address. That's what was > in the arp table. If they're following the standards, only one of PE1 > and PE2 will accept packets with that destination mac address. The > other, recognizing that the packet is not addressed to it, drops it. There are many reasons why practical devices (such as VXR) don't use MAC HW filters. Such as your PHY runs out of HW filter slots, or the HW does not support per-vlan HW filter, or there is 1 subinterface with EoMPLS configured or other type of L2 service, requiring reception of any DMAC. There are also many reasons why both routers have the DMAC in their HW filter, such as VRRP, HSRP. > for example. But then this ring configuration doesn't exist in an AWS > VPC and I've not particularly observed a lot of packet duplication out > of Amazon. Amazon does nothing standard, it's all AMZN. Hope this helps! -- ++ytti
Re: Is there any data on packet duplication?
On Mon, Jun 22, 2020 at 10:21 PM Saku Ytti wrote: > On Tue, 23 Jun 2020 at 08:12, William Herrin wrote: > > That's what spanning tree and its compatriots are for. Otherwise, > > ordinary broadcast traffic (like those arp packets) would travel in a > > loop, flooding the network and it would just about instantly collapse > > when you first turned it on. > > Metro: S1-S2-S3-S1 > PE1: S1 > PE2: S2 > Customer: S3 > STP blocking: ANY > > S3 sends frame, it is unknown unicast flooded, S1+S2 both get it > (regardless of which metro port blocks), which will send it via PE to > Internet. There's a link in the chain you haven't explained. The packet which entered at S3 has a unicast destination MAC address. That's what was in the arp table. If they're following the standards, only one of PE1 and PE2 will accept packets with that destination mac address. The other, recognizing that the packet is not addressed to it, drops it. Recall that ethernet worked without duplicating packets back in the days of hubs when all stations received all packets. This is how. That having been said, I've seen vendors creatively breach the boundary between L2 and L3 with some really peculiar results. AWS VPCs for example. But then this ring configuration doesn't exist in an AWS VPC and I've not particularly observed a lot of packet duplication out of Amazon. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: Is there any data on packet duplication?
On Tue, 23 Jun 2020 at 09:32, Sabri Berisha wrote: > Aaah yes, fair point! Thanks $deity for default timers that make no sense. Add low-traffic connection and default 1024s maxPoll of NTP and this duplication is guaranteed to happen for 97.9% of packets. -- ++ytti
Re: Is there any data on packet duplication?
- On Jun 22, 2020, at 11:21 PM, Saku Ytti s...@ytti.fi wrote: Hi Saku, > On Tue, 23 Jun 2020 at 09:15, Sabri Berisha wrote: > >> Yeah, except that unless you use static ARP entries, I can't come up >> with a plausible scenario in which this would happen for NTP. Assuming >> we're talking about a non-local NTP server, S3 will not send an NTP >> packet without first sending an ARP. Yes, your ARP will be flooded, >> but your NTP packet won't be transmitted until there is an ARP reply. >> By that time MACs have been learned, and the NTP packet will not be >> considered BUM traffic, right? > > The plausible scenario is the one I explained. The crucial detail is > MAC timeout (catalyst 300s) being shorter than ARP timeout (cisco 4h). > So the device generating the packet knows the MAC address, the L2 does > not. Aaah yes, fair point! Thanks $deity for default timers that make no sense. Thanks, Sabri
Re: Is there any data on packet duplication?
On Tue, 23 Jun 2020 at 09:15, Sabri Berisha wrote: > Yeah, except that unless you use static ARP entries, I can't come up > with a plausible scenario in which this would happen for NTP. Assuming > we're talking about a non-local NTP server, S3 will not send an NTP > packet without first sending an ARP. Yes, your ARP will be flooded, > but your NTP packet won't be transmitted until there is an ARP reply. > By that time MACs have been learned, and the NTP packet will not be > considered BUM traffic, right? The plausible scenario is the one I explained. The crucial detail is MAC timeout (catalyst 300s) being shorter than ARP timeout (cisco 4h). So the device generating the packet knows the MAC address, the L2 does not. Hope this helps! -- ++ytti
Re: Is there any data on packet duplication?
- On Jun 22, 2020, at 10:21 PM, Saku Ytti s...@ytti.fi wrote: Hi, > Metro: S1-S2-S3-S1 > PE1: S1 > PE2: S2 > Customer: S3 > STP blocking: ANY > > S3 sends frame, it is unknown unicast flooded, S1+S2 both get it > (regardless of which metro port blocks), which will send it via PE to > Internet. > > STP doesn't help, at all. Hope this helps. Yeah, except that unless you use static ARP entries, I can't come up with a plausible scenario in which this would happen for NTP. Assuming we're talking about a non-local NTP server, S3 will not send an NTP packet without first sending an ARP. Yes, your ARP will be flooded, but your NTP packet won't be transmitted until there is an ARP reply. By that time MACs have been learned, and the NTP packet will not be considered BUM traffic, right? That said, I have seen packet duplication in L2 onlu networks that I've worked on myself, but that was because I disregarded a lot of rules from the imaginary networking handbook. Thanks, Sabri
Re: Is there any data on packet duplication?
On 23/Jun/20 07:52, Saku Ytti wrote: > > S1-S2-S3-S1 is operator L2 metro-ring, which connects customers and > 2xPE routers. It VLAN backhauls customers to PE. Okay. In 2014, we hit a similar issue, although not in a ring. Our previous architecture was to interconnect edge routers via downstream, interconnected aggregation switches to which customers connected in order to support VRRP. Since customers do strange things, both edge routers received the same traffic, which caused pain. Since then, we don't support VRRP for customers any longer, nor do we interconnect aggregation switches that map to different edge routers. Your example scenario describes what we experienced back then. Mark.
