Id like to think some lowly janitor was picked up from his vacation by a
helicopter right from his john boat on the lake, whisked to an airport and
loaded onto a leer jet to california, then police escorted to FB HQ to pull
out his big ol key ring and open the door
On Tue, Oct 5, 2021 at 4:12 PM Chuck McCown via AF <af@af.afmug.com> wrote:
> Probably more like some kind of automation with a bug in it. I got a kick
> out of FB actually saying that this caused some of their people to be
> physically locked out of data center equipment they needed to access to fix
> it. I don’t think I would have ever admitted that.
>
> *From:* Nate Burke
> *Sent:* Tuesday, October 5, 2021 2:45 PM
> *To:* AnimalFarm Microwave Users Group
> *Subject:* Re: [AFMUG] facebook outage
>
> I think the more plausible explanation is that Zuckerberg was in a bar
> with some guys he was trying to impress saying something like 'I do SO run
> Facebook' and then whips out his phone and instead of deleting their
> profile, he presses the 'delete all' button by accident.
>
> I've been at some small businesses where they can't change an Ethernet
> port speed without a 3 person peer review of the config update, I have
> trouble believing that one guy mistyped something.
>
> On 10/5/2021 12:17 PM, Steve Jones wrote:
>
> ill bet whatever their process was, its not what it is and delayed
> commits/fallbacks are instituted forthwith. Not by the guy who committed
> this set of changes though, hes at the bottom of a river now
>
> On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson <c...@infowest.com>
> wrote:
>
>> I wonder if they did a "commit confirmed 30”? If not, would that have
>> avoided the whole thing?
>>
>>
>> On Oct 5, 2021, at 11:08 AM, Carl Peterson <cpeter...@portnetworks.com>
>> wrote:
>>
>> "an engineer put in a config change on a BGP router that propagated to
>> the other BGP routers and cut off access"
>>
>> Sounds like something I would do which is why I generally insist on
>> having a remote tech physically in the building with a console cable if I
>> don't have OOB console access when changing BGP on my edge.
>>
>> On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com> wrote:
>>
>>> FB has posted that an engineer put in a config change on a BGP router
>>> that propagated to the other BGP routers and cut off access. Sounds like
>>> it took a while to track down the config change. I do know someone who is
>>> intimate with that group but he's not talking... Drinking, but not
>>> talking...
>>>
>>> On 10/5/21 9:17 AM, Bill Prince wrote:
>>>
>>> I see it differently. In the beginning cloudflare thought something was
>>> wrong with their systems. It doesn't explain what happened inside facebook,
>>> probably because only facebook knows what went wrong. That's not
>>> cloudflare's problem. We may never know what happened at facebook probably
>>> because it's very embarrassing to facebook.
>>>
>>>
>>>
>>> bp
>>> <part15sbs{at}gmail{dot}com>
>>>
>>> On 10/5/2021 9:01 AM, Robert wrote:
>>>
>>> I really don't like that cloudflare posted this, as it doesn't do
>>> anything to tell why. It's not a root cause, it's an effect and cloudflare
>>> looks like they are just trying to say how smart they are.
>>>
>>> On 10/5/21 8:40 AM, Bill Prince wrote:
>>>
>>>
>>>
>>> https://blog.cloudflare.com/october-2021-facebook-outage/
>>>
>>>
>>>
>>> bp
>>> <part15sbs{at}gmail{dot}com>
>>>
>>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>>>
>>> Sign him up for a SNAP card.
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 5, 2021, at 8:25 AM, Bill Prince mailto:part15...@gmail.com
>>> wrote:
>>>
>>>
>>>
>>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
>>> yesterday. Now he's only worth around $121 billion.
>>>
>>>
>>>
>>> bp
>>> <part15sbs{at}gmail{dot}com>
>>>
>>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>>>
>>> I am very curious what it was that took 6 hours to figure out or undo.
>>>
>>>
>>>
>>> …..but I wouldn’t judge too harshly. When any contraption gets big and
>>> complex it’s easy to overlook one seemingly small thing.
>>>
>>>
>>>
>>> Remember this XKCD?
>>>
>>>
>>> <image001.png>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* AF mailto:af-boun...@af.afmug.com *On Behalf Of *Steve Jones
>>> *Sent:* Monday, October 04, 2021 6:51 PM
>>> *To:* AnimalFarm Microwave Users Group mailto:af@af.afmug.com
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>> so it turns out FB murdered the announcements for the DNS server
>>> prefixes, including the registrar.
>>>
>>>
>>> <https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ>
>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>>
>>>
>>>
>>> This really seems like an elementary design flaw. Even a moron like me
>>> knows not to poke the DNS subnets all at once, and we probably get way less
>>> than 1k daily lookups across all the domains we host DNS for. Ill bet
>>> nobody involved in this makes less than 80k a year. facebook lost 4% stock
>>> value today attributed to this on top of the almost percent attributed to
>>> the "whistleblower". I get nervous when I lose the company a number 10
>>> wrench.
>>>
>>>
>>>
>>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <
>>> par...@cyberbroadband.net> wrote:
>>>
>>>
>>>
>>> it is alive here again
>>>
>>>
>>>
>>> ----- Original Message -----
>>>
>>> *From:* CBB - Jay Fuller
>>>
>>> *To:* AnimalFarm Microwave Users Group
>>>
>>> *Sent:* Monday, October 4, 2021 4:44 PM
>>>
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>>
>>>
>>> yup. resolving here. but nothing coming up yet.
>>>
>>>
>>>
>>> ----- Original Message -----
>>>
>>> *From:* Mike Hammett
>>>
>>> *To:* AnimalFarm Microwave Users Group
>>>
>>> *Sent:* Monday, October 4, 2021 4:36 PM
>>>
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>> They're back... ish.
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions <http://www.ics-il.com/>
>>> <https://www.facebook.com/ICSIL>
>>> <~WRD0002.jpg> <https://www.facebook.com/ICSIL>
>>> <~WRD0002.jpg>
>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>>> <~WRD0002.jpg>
>>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>>> <~WRD0002.jpg>
>>> <https://twitter.com/ICSIL>
>>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>>> <~WRD0002.jpg>
>>> <https://www.facebook.com/mdwestix>
>>> <~WRD0002.jpg>
>>> <https://www.linkedin.com/company/midwest-internet-exchange>
>>> <~WRD0002.jpg>
>>> <https://twitter.com/mdwestix>
>>> The Brothers WISP <http://www.thebrotherswisp.com/>
>>> <~WRD0002.jpg>
>>> <https://www.facebook.com/thebrotherswisp>
>>> <~WRD0002.jpg>
>>>
>>>
>>>
>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>> ------------------------------
>>>
>>> *From: *"CBB - Jay Fuller" <par...@cyberbroadband.net>
>>> *To: *af@af.afmug.com
>>> *Sent: *Monday, October 4, 2021 1:13:01 PM
>>> *Subject: *[AFMUG] facebook outage
>>>
>>>
>>>
>>> anyone else feeling alone and isolated without facebook? I need my safe
>>> space (no, not myspace) back.
>>>
>>>
>>>
>>> babaahahah.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> AF mailing list
>>> AF@af.afmug.com
>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com>
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>
>>>
>>> ------------------------------
>>>
>>> --
>>> AF mailing list
>>> AF@af.afmug.com
>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com>
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>
>>> ------------------------------
>>>
>>> --
>>> AF mailing list
>>> AF@af.afmug.com
>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com>
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>
>>> --
>>> AF mailing list
>>> AF@af.afmug.com
>>> <http://af.afmug.com/mailman/listinfo/af_af.afmug.com>
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>
>>>
>>> --
>>> AF mailing list
>>> AF@af.afmug.com
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> AF mailing list
>>> AF@af.afmug.com
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>
>>
>>
>> --
>>
>> Carl Peterson
>>
>> *PORT NETWORKS*
>>
>> 401 E Pratt St, Ste 2553
>>
>> Baltimore, MD 21202
>>
>> (410) 637-3707
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>
>
>
> ------------------------------
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
