Nobody who carries an angle grinder or a sawsall is ever physically locked out. Sounds like lack of preparation really.
From: AF <af-boun...@af.afmug.com> On Behalf Of Chuck McCown via AF Sent: Tuesday, October 05, 2021 5:12 PM To: AnimalFarm Microwave Users Group <af@af.afmug.com> Cc: Chuck McCown <ch...@go-mtc.com> Subject: Re: [AFMUG] facebook outage Probably more like some kind of automation with a bug in it. I got a kick out of FB actually saying that this caused some of their people to be physically locked out of data center equipment they needed to access to fix it. I don’t think I would have ever admitted that. From: Nate Burke Sent: Tuesday, October 5, 2021 2:45 PM To: AnimalFarm Microwave Users Group Subject: Re: [AFMUG] facebook outage I think the more plausible explanation is that Zuckerberg was in a bar with some guys he was trying to impress saying something like 'I do SO run Facebook' and then whips out his phone and instead of deleting their profile, he presses the 'delete all' button by accident. I've been at some small businesses where they can't change an Ethernet port speed without a 3 person peer review of the config update, I have trouble believing that one guy mistyped something. On 10/5/2021 12:17 PM, Steve Jones wrote: ill bet whatever their process was, its not what it is and delayed commits/fallbacks are instituted forthwith. Not by the guy who committed this set of changes though, hes at the bottom of a river now On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson <c...@infowest.com <mailto:c...@infowest.com> > wrote: I wonder if they did a "commit confirmed 30”? If not, would that have avoided the whole thing? On Oct 5, 2021, at 11:08 AM, Carl Peterson <cpeter...@portnetworks.com <mailto:cpeter...@portnetworks.com> > wrote: "an engineer put in a config change on a BGP router that propagated to the other BGP routers and cut off access" Sounds like something I would do which is why I generally insist on having a remote tech physically in the building with a console cable if I don't have OOB console access when changing BGP on my edge. On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com <mailto:i...@avantwireless.com> > wrote: FB has posted that an engineer put in a config change on a BGP router that propagated to the other BGP routers and cut off access. Sounds like it took a while to track down the config change. I do know someone who is intimate with that group but he's not talking... Drinking, but not talking... On 10/5/21 9:17 AM, Bill Prince wrote: I see it differently. In the beginning cloudflare thought something was wrong with their systems. It doesn't explain what happened inside facebook, probably because only facebook knows what went wrong. That's not cloudflare's problem. We may never know what happened at facebook probably because it's very embarrassing to facebook. bp <part15sbs{at}gmail{dot}com> On 10/5/2021 9:01 AM, Robert wrote: I really don't like that cloudflare posted this, as it doesn't do anything to tell why. It's not a root cause, it's an effect and cloudflare looks like they are just trying to say how smart they are. On 10/5/21 8:40 AM, Bill Prince wrote: https://blog.cloudflare.com/october-2021-facebook-outage/ bp <part15sbs{at}gmail{dot}com> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote: Sign him up for a SNAP card. Sent from my iPhone On Oct 5, 2021, at 8:25 AM, Bill Prince mailto:part15...@gmail.com wrote: Heard the Zuch got around $6 or $7 billion trimmed from his net worth yesterday. Now he's only worth around $121 billion. bp <part15sbs{at}gmail{dot}com> On 10/5/2021 6:52 AM, dmmoff...@gmail.com <mailto:dmmoff...@gmail.com> wrote: I am very curious what it was that took 6 hours to figure out or undo. …..but I wouldn’t judge too harshly. When any contraption gets big and complex it’s easy to overlook one seemingly small thing. Remember this XKCD? <image001.png> From: AF mailto:af-boun...@af.afmug.com On Behalf Of Steve Jones Sent: Monday, October 04, 2021 6:51 PM To: AnimalFarm Microwave Users Group mailto:af@af.afmug.com Subject: Re: [AFMUG] facebook outage so it turns out FB murdered the announcements for the DNS server prefixes, including the registrar. https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ This really seems like an elementary design flaw. Even a moron like me knows not to poke the DNS subnets all at once, and we probably get way less than 1k daily lookups across all the domains we host DNS for. Ill bet nobody involved in this makes less than 80k a year. facebook lost 4% stock value today attributed to this on top of the almost percent attributed to the "whistleblower". I get nervous when I lose the company a number 10 wrench. On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <par...@cyberbroadband.net <mailto:par...@cyberbroadband.net> > wrote: it is alive here again ----- Original Message ----- From: CBB - Jay Fuller To: AnimalFarm Microwave Users Group Sent: Monday, October 4, 2021 4:44 PM Subject: Re: [AFMUG] facebook outage yup. resolving here. but nothing coming up yet. ----- Original Message ----- From: Mike Hammett To: AnimalFarm Microwave Users Group Sent: Monday, October 4, 2021 4:36 PM Subject: Re: [AFMUG] facebook outage They're back... ish. ----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <~WRD0002.jpg> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <~WRD0002.jpg> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://www.linkedin.com/company/intelligent-computing-solutions> <~WRD0002.jpg> <https://twitter.com/ICSIL> <https://twitter.com/ICSIL> <~WRD0002.jpg> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.facebook.com/mdwestix> <~WRD0002.jpg> <https://www.linkedin.com/company/midwest-internet-exchange> <https://www.linkedin.com/company/midwest-internet-exchange> <~WRD0002.jpg> <https://twitter.com/mdwestix> <https://twitter.com/mdwestix> <~WRD0002.jpg> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.facebook.com/thebrotherswisp> <~WRD0002.jpg> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> <~WRD0002.jpg> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> _____ From: "CBB - Jay Fuller" <par...@cyberbroadband.net <mailto:par...@cyberbroadband.net> > To: af@af.afmug.com <mailto:af@af.afmug.com> Sent: Monday, October 4, 2021 1:13:01 PM Subject: [AFMUG] facebook outage anyone else feeling alone and isolated without facebook? I need my safe space (no, not myspace) back. babaahahah. -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com _____ -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com _____ -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- Carl Peterson PORT NETWORKS 401 E Pratt St, Ste 2553 Baltimore, MD 21202 (410) 637-3707 -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com _____ -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
