If 30 minutes would have been long enough. I wonder how long it really takes
for BGP changes to propagate.
Having contracted for FB before, let me tell you their management style is
horrible. Conference calls to plan conference calls. I got my ass chewed for
being honest on a call. Can’t cause shame or embarrassment. They had an
engineering company that royally screwed things up but I had to act like they
were the best company on the planet during calls. Lots of infighting between
different FB departments. Lots of silos. The managers are imperious.
In any event, I am betting they are in the process of convening a NASA type
challenger explosion panel to have a year’s worth of conference calls to make
even more bureaucracy to prevent something like this from happening again.
Back in olden times, when everyone that owned a router trusted everyone else,
someone announced they could take all IPs over BGP. I think they were in
Champaign Urbana. I was in Quincy Ill at the time. In any event it brought
our part of the internet to a screeching halt. I think there was an RFCs
created that very day and solutions created in just a few more days over that
one.
From: Cassidy B. Larson
Sent: Tuesday, October 5, 2021 11:12 AM
To: AnimalFarm Microwave Users Group
Subject: Re: [AFMUG] facebook outage
I wonder if they did a "commit confirmed 30”? If not, would that have avoided
the whole thing?
On Oct 5, 2021, at 11:08 AM, Carl Peterson <cpeter...@portnetworks.com> wrote:
"an engineer put in a config change on a BGP router that propagated to the
other BGP routers and cut off access"
Sounds like something I would do which is why I generally insist on having a
remote tech physically in the building with a console cable if I don't have OOB
console access when changing BGP on my edge.
On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com> wrote:
FB has posted that an engineer put in a config change on a BGP router that
propagated to the other BGP routers and cut off access. Sounds like it took a
while to track down the config change. I do know someone who is intimate with
that group but he's not talking... Drinking, but not talking...
On 10/5/21 9:17 AM, Bill Prince wrote:
I see it differently. In the beginning cloudflare thought something was
wrong with their systems. It doesn't explain what happened inside facebook,
probably because only facebook knows what went wrong. That's not cloudflare's
problem. We may never know what happened at facebook probably because it's very
embarrassing to facebook.
bp
<part15sbs{at}gmail{dot}com>On 10/5/2021 9:01 AM, Robert wrote:
I really don't like that cloudflare posted this, as it doesn't do
anything to tell why. It's not a root cause, it's an effect and cloudflare
looks like they are just trying to say how smart they are.
On 10/5/21 8:40 AM, Bill Prince wrote:
https://blog.cloudflare.com/october-2021-facebook-outage/
bp
<part15sbs{at}gmail{dot}com>On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
Sign him up for a SNAP card.
Sent from my iPhone
On Oct 5, 2021, at 8:25 AM, Bill Prince
mailto:part15...@gmail.com wrote:
Heard the Zuch got around $6 or $7 billion trimmed from his net
worth yesterday. Now he's only worth around $121 billion.
bp
<part15sbs{at}gmail{dot}com>On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
I am very curious what it was that took 6 hours to figure out
or undo.
…..but I wouldn’t judge too harshly. When any contraption gets
big and complex it’s easy to overlook one seemingly small thing.
Remember this XKCD?
<image001.png>
From: AF mailto:af-boun...@af.afmug.com On Behalf Of Steve Jones
Sent: Monday, October 04, 2021 6:51 PM
To: AnimalFarm Microwave Users Group mailto:af@af.afmug.com
Subject: Re: [AFMUG] facebook outage
so it turns out FB murdered the announcements for the DNS
server prefixes, including the registrar.
https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
This really seems like an elementary design flaw. Even a moron
like me knows not to poke the DNS subnets all at once, and we probably get way
less than 1k daily lookups across all the domains we host DNS for. Ill bet
nobody involved in this makes less than 80k a year. facebook lost 4% stock
value today attributed to this on top of the almost percent attributed to the
"whistleblower". I get nervous when I lose the company a number 10 wrench.
On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller
<par...@cyberbroadband.net> wrote:
it is alive here again
----- Original Message -----
From: CBB - Jay Fuller
To: AnimalFarm Microwave Users Group
Sent: Monday, October 4, 2021 4:44 PM
Subject: Re: [AFMUG] facebook outage
yup. resolving here. but nothing coming up yet.
----- Original Message -----
From: Mike Hammett
To: AnimalFarm Microwave Users Group
Sent: Monday, October 4, 2021 4:36 PM
Subject: Re: [AFMUG] facebook outage
They're back... ish.
-----
Mike Hammett
Intelligent Computing Solutions
<~WRD0002.jpg>
<~WRD0002.jpg>
<~WRD0002.jpg>
<~WRD0002.jpg>
Midwest Internet Exchange
<~WRD0002.jpg>
<~WRD0002.jpg>
<~WRD0002.jpg>
The Brothers WISP
<~WRD0002.jpg>
<~WRD0002.jpg>
----------------------------------------------------------
From: "CBB - Jay Fuller" <par...@cyberbroadband.net>
To: af@af.afmug.com
Sent: Monday, October 4, 2021 1:13:01 PM
Subject: [AFMUG] facebook outage
anyone else feeling alone and isolated without facebook?
I need my safe space (no, not myspace) back.
babaahahah.
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
----------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
Carl Peterson
PORT NETWORKS
401 E Pratt St, Ste 2553
Baltimore, MD 21202
(410) 637-3707
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--------------------------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
