Probably more like some kind of automation with a bug in it. I got a kick out
of FB actually saying that this caused some of their people to be physically
locked out of data center equipment they needed to access to fix it. I don’t
think I would have ever admitted that.
From: Nate Burke
Sent: Tuesday, October 5, 2021 2:45 PM
To: AnimalFarm Microwave Users Group
Subject: Re: [AFMUG] facebook outage
I think the more plausible explanation is that Zuckerberg was in a bar with
some guys he was trying to impress saying something like 'I do SO run Facebook'
and then whips out his phone and instead of deleting their profile, he presses
the 'delete all' button by accident.
I've been at some small businesses where they can't change an Ethernet port
speed without a 3 person peer review of the config update, I have trouble
believing that one guy mistyped something.
On 10/5/2021 12:17 PM, Steve Jones wrote:
ill bet whatever their process was, its not what it is and delayed
commits/fallbacks are instituted forthwith. Not by the guy who committed this
set of changes though, hes at the bottom of a river now
On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson <c...@infowest.com> wrote:
I wonder if they did a "commit confirmed 30”? If not, would that have
avoided the whole thing?
On Oct 5, 2021, at 11:08 AM, Carl Peterson <cpeter...@portnetworks.com>
wrote:
"an engineer put in a config change on a BGP router that propagated to
the other BGP routers and cut off access"
Sounds like something I would do which is why I generally insist on
having a remote tech physically in the building with a console cable if I don't
have OOB console access when changing BGP on my edge.
On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com> wrote:
FB has posted that an engineer put in a config change on a BGP router
that propagated to the other BGP routers and cut off access. Sounds like it
took a while to track down the config change. I do know someone who is
intimate with that group but he's not talking... Drinking, but not talking...
On 10/5/21 9:17 AM, Bill Prince wrote:
I see it differently. In the beginning cloudflare thought something
was wrong with their systems. It doesn't explain what happened inside facebook,
probably because only facebook knows what went wrong. That's not cloudflare's
problem. We may never know what happened at facebook probably because it's very
embarrassing to facebook.
bp
<part15sbs{at}gmail{dot}com>On 10/5/2021 9:01 AM, Robert wrote:
I really don't like that cloudflare posted this, as it doesn't do
anything to tell why. It's not a root cause, it's an effect and cloudflare
looks like they are just trying to say how smart they are.
On 10/5/21 8:40 AM, Bill Prince wrote:
https://blog.cloudflare.com/october-2021-facebook-outage/
bp
<part15sbs{at}gmail{dot}com>On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
Sign him up for a SNAP card.
Sent from my iPhone
On Oct 5, 2021, at 8:25 AM, Bill Prince
mailto:part15...@gmail.com wrote:
Heard the Zuch got around $6 or $7 billion trimmed from his
net worth yesterday. Now he's only worth around $121 billion.
bp
<part15sbs{at}gmail{dot}com>On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
I am very curious what it was that took 6 hours to figure
out or undo.
…..but I wouldn’t judge too harshly. When any contraption
gets big and complex it’s easy to overlook one seemingly small thing.
Remember this XKCD?
<image001.png>
From: AF mailto:af-boun...@af.afmug.com On Behalf Of Steve
Jones
Sent: Monday, October 04, 2021 6:51 PM
To: AnimalFarm Microwave Users Group mailto:af@af.afmug.com
Subject: Re: [AFMUG] facebook outage
so it turns out FB murdered the announcements for the DNS
server prefixes, including the registrar.
https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
This really seems like an elementary design flaw. Even a
moron like me knows not to poke the DNS subnets all at once, and we probably
get way less than 1k daily lookups across all the domains we host DNS for. Ill
bet nobody involved in this makes less than 80k a year. facebook lost 4% stock
value today attributed to this on top of the almost percent attributed to the
"whistleblower". I get nervous when I lose the company a number 10 wrench.
On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller
<par...@cyberbroadband.net> wrote:
it is alive here again
----- Original Message -----
From: CBB - Jay Fuller
To: AnimalFarm Microwave Users Group
Sent: Monday, October 4, 2021 4:44 PM
Subject: Re: [AFMUG] facebook outage
yup. resolving here. but nothing coming up yet.
----- Original Message -----
From: Mike Hammett
To: AnimalFarm Microwave Users Group
Sent: Monday, October 4, 2021 4:36 PM
Subject: Re: [AFMUG] facebook outage
They're back... ish.
-----
Mike Hammett
Intelligent Computing Solutions
<~WRD0002.jpg>
<~WRD0002.jpg>
<~WRD0002.jpg>
<~WRD0002.jpg>
Midwest Internet Exchange
<~WRD0002.jpg>
<~WRD0002.jpg>
<~WRD0002.jpg>
The Brothers WISP
<~WRD0002.jpg>
<~WRD0002.jpg>
------------------------------------------------------
From: "CBB - Jay Fuller" <par...@cyberbroadband.net>
To: af@af.afmug.com
Sent: Monday, October 4, 2021 1:13:01 PM
Subject: [AFMUG] facebook outage
anyone else feeling alone and isolated without
facebook? I need my safe space (no, not myspace) back.
babaahahah.
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
Carl Peterson
PORT NETWORKS
401 E Pratt St, Ste 2553
Baltimore, MD 21202
(410) 637-3707
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--------------------------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
