I wonder if they did a "commit confirmed 30”? If not, would that have avoided the whole thing?
> On Oct 5, 2021, at 11:08 AM, Carl Peterson <cpeter...@portnetworks.com> wrote: > > "an engineer put in a config change on a BGP router that propagated to the > other BGP routers and cut off access" > > Sounds like something I would do which is why I generally insist on having a > remote tech physically in the building with a console cable if I don't have > OOB console access when changing BGP on my edge. > > On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com > <mailto:i...@avantwireless.com>> wrote: > FB has posted that an engineer put in a config change on a BGP router that > propagated to the other BGP routers and cut off access. Sounds like it took > a while to track down the config change. I do know someone who is intimate > with that group but he's not talking... Drinking, but not talking... > > On 10/5/21 9:17 AM, Bill Prince wrote: >> I see it differently. In the beginning cloudflare thought something was >> wrong with their systems. It doesn't explain what happened inside facebook, >> probably because only facebook knows what went wrong. That's not >> cloudflare's problem. We may never know what happened at facebook probably >> because it's very embarrassing to facebook. >> >> >> >> bp >> <part15sbs{at}gmail{dot}com> >> On 10/5/2021 9:01 AM, Robert wrote: >>> I really don't like that cloudflare posted this, as it doesn't do anything >>> to tell why. It's not a root cause, it's an effect and cloudflare looks >>> like they are just trying to say how smart they are. >>> >>> On 10/5/21 8:40 AM, Bill Prince wrote: >>>> >>>> https://blog.cloudflare.com/october-2021-facebook-outage/ >>>> <https://blog.cloudflare.com/october-2021-facebook-outage/> >>>> >>>> bp >>>> <part15sbs{at}gmail{dot}com> >>>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote: >>>>> Sign him up for a SNAP card. >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Oct 5, 2021, at 8:25 AM, Bill Prince <part15...@gmail.com> >>>>>> <mailto:part15...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth >>>>>> yesterday. Now he's only worth around $121 billion. >>>>>> >>>>>> >>>>>> >>>>>> bp >>>>>> <part15sbs{at}gmail{dot}com> >>>>>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com <mailto:dmmoff...@gmail.com> >>>>>> wrote: >>>>>>> I am very curious what it was that took 6 hours to figure out or undo. >>>>>>> >>>>>>> >>>>>>> >>>>>>> …..but I wouldn’t judge too harshly. When any contraption gets big and >>>>>>> complex it’s easy to overlook one seemingly small thing. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Remember this XKCD? >>>>>>> >>>>>>> >>>>>>> >>>>>>> <image001.png> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: AF <af-boun...@af.afmug.com> <mailto:af-boun...@af.afmug.com> On >>>>>>> Behalf Of Steve Jones >>>>>>> Sent: Monday, October 04, 2021 6:51 PM >>>>>>> To: AnimalFarm Microwave Users Group <af@af.afmug.com> >>>>>>> <mailto:af@af.afmug.com> >>>>>>> Subject: Re: [AFMUG] facebook outage >>>>>>> >>>>>>> >>>>>>> >>>>>>> so it turns out FB murdered the announcements for the DNS server >>>>>>> prefixes, including the registrar. >>>>>>> >>>>>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ >>>>>>> >>>>>>> <https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ> >>>>>>> >>>>>>> >>>>>>> This really seems like an elementary design flaw. Even a moron like me >>>>>>> knows not to poke the DNS subnets all at once, and we probably get way >>>>>>> less than 1k daily lookups across all the domains we host DNS for. Ill >>>>>>> bet nobody involved in this makes less than 80k a year. facebook lost >>>>>>> 4% stock value today attributed to this on top of the almost percent >>>>>>> attributed to the "whistleblower". I get nervous when I lose the >>>>>>> company a number 10 wrench. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller >>>>>>> <par...@cyberbroadband.net <mailto:par...@cyberbroadband.net>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> it is alive here again >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> >>>>>>> From: CBB - Jay Fuller <mailto:par...@cyberbroadband.net> >>>>>>> To: AnimalFarm Microwave Users Group <mailto:af@af.afmug.com> >>>>>>> Sent: Monday, October 4, 2021 4:44 PM >>>>>>> >>>>>>> Subject: Re: [AFMUG] facebook outage >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> yup. resolving here. but nothing coming up yet. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> >>>>>>> From: Mike Hammett <mailto:af...@ics-il.net> >>>>>>> To: AnimalFarm Microwave Users Group <mailto:af@af.afmug.com> >>>>>>> Sent: Monday, October 4, 2021 4:36 PM >>>>>>> >>>>>>> Subject: Re: [AFMUG] facebook outage >>>>>>> >>>>>>> >>>>>>> >>>>>>> They're back... ish. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- >>>>>>> Mike Hammett >>>>>>> Intelligent Computing Solutions <http://www.ics-il.com/> >>>>>>> <https://www.facebook.com/ICSIL> >>>>>>> <~WRD0002.jpg> <https://www.facebook.com/ICSIL><~WRD0002.jpg> >>>>>>> >>>>>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb><~WRD0002.jpg> >>>>>>> >>>>>>> <https://www.linkedin.com/company/intelligent-computing-solutions><~WRD0002.jpg> >>>>>>> <https://twitter.com/ICSIL> >>>>>>> Midwest Internet Exchange <http://www.midwest-ix.com/> >>>>>>> <~WRD0002.jpg> >>>>>>> <https://www.facebook.com/mdwestix><~WRD0002.jpg> >>>>>>> >>>>>>> <https://www.linkedin.com/company/midwest-internet-exchange><~WRD0002.jpg> >>>>>>> <https://twitter.com/mdwestix> >>>>>>> The Brothers WISP <http://www.thebrotherswisp.com/> >>>>>>> <~WRD0002.jpg> >>>>>>> <https://www.facebook.com/thebrotherswisp><~WRD0002.jpg> >>>>>>> >>>>>>> >>>>>>> >>>>>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> >>>>>>> From: "CBB - Jay Fuller" <par...@cyberbroadband.net >>>>>>> <mailto:par...@cyberbroadband.net>> >>>>>>> To: af@af.afmug.com <mailto:af@af.afmug.com> >>>>>>> Sent: Monday, October 4, 2021 1:13:01 PM >>>>>>> Subject: [AFMUG] facebook outage >>>>>>> >>>>>>> >>>>>>> >>>>>>> anyone else feeling alone and isolated without facebook? I need my >>>>>>> safe space (no, not myspace) back. >>>>>>> >>>>>>> >>>>>>> >>>>>>> babaahahah. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> AF mailing list >>>>>>> AF@af.afmug.com <mailto:AF@af.afmug.com> >>>>>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>>>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> AF mailing list >>>>>>> AF@af.afmug.com <mailto:AF@af.afmug.com> >>>>>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>>>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com> >>>>>>> -- >>>>>>> AF mailing list >>>>>>> AF@af.afmug.com <mailto:AF@af.afmug.com> >>>>>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>>>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com> >>>>>>> -- >>>>>>> AF mailing list >>>>>>> AF@af.afmug.com <mailto:AF@af.afmug.com> >>>>>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>>>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com> >>>>>> -- >>>>>> AF mailing list >>>>>> AF@af.afmug.com <mailto:AF@af.afmug.com> >>>>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com >>>>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com> >>>>> >>>> >>> >>> >> > > -- > AF mailing list > AF@af.afmug.com <mailto:AF@af.afmug.com> > http://af.afmug.com/mailman/listinfo/af_af.afmug.com > <http://af.afmug.com/mailman/listinfo/af_af.afmug.com> > > > -- > Carl Peterson > > PORT NETWORKS > > 401 E Pratt St, Ste 2553 > > Baltimore, MD 21202 > > (410) 637-3707 > > -- > AF mailing list > AF@af.afmug.com > http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
