No, they just had to find a laptop that had a serial port on it. They
probably only are issued company tablets.
On 10/5/2021 4:22 PM, Steve Jones wrote:
Id like to think some lowly janitor was picked up from his vacation by
a helicopter right from his john boat on the lake, whisked to an
airport and loaded onto a leer jet to california, then police escorted
to FB HQ to pull out his big ol key ring and open the door
On Tue, Oct 5, 2021 at 4:12 PM Chuck McCown via AF <af@af.afmug.com
<mailto:af@af.afmug.com>> wrote:
Probably more like some kind of automation with a bug in it. I
got a kick out of FB actually saying that this caused some of
their people to be physically locked out of data center equipment
they needed to access to fix it. I don’t think I would have ever
admitted that.
*From:* Nate Burke
*Sent:* Tuesday, October 5, 2021 2:45 PM
*To:* AnimalFarm Microwave Users Group
*Subject:* Re: [AFMUG] facebook outage
I think the more plausible explanation is that Zuckerberg was in a
bar with some guys he was trying to impress saying something like
'I do SO run Facebook' and then whips out his phone and instead of
deleting their profile, he presses the 'delete all' button by
accident.
I've been at some small businesses where they can't change an
Ethernet port speed without a 3 person peer review of the config
update, I have trouble believing that one guy mistyped something.
On 10/5/2021 12:17 PM, Steve Jones wrote:
ill bet whatever their process was, its not what it is and
delayed commits/fallbacks are instituted forthwith. Not by the
guy who committed this set of changes though, hes at the bottom
of a river now
On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson
<c...@infowest.com> wrote:
I wonder if they did a "commit confirmed 30”? If not, would
that have avoided the whole thing?
On Oct 5, 2021, at 11:08 AM, Carl Peterson
<cpeter...@portnetworks.com> wrote:
"an engineer put in a config change on a BGP router that
propagated to the other BGP routers and cut off access"
Sounds like something I would do which is why I generally
insist on having a remote tech physically in the building
with a console cable if I don't have OOB console access when
changing BGP on my edge.
On Tue, Oct 5, 2021 at 11:34 AM Robert
<i...@avantwireless.com> wrote:
FB has posted that an engineer put in a config change on
a BGP router that propagated to the other BGP routers
and cut off access. Sounds like it took a while to
track down the config change. I do know someone who is
intimate with that group but he's not talking...
Drinking, but not talking...
On 10/5/21 9:17 AM, Bill Prince wrote:
I see it differently. In the beginning cloudflare
thought something was wrong with their systems. It
doesn't explain what happened inside facebook, probably
because only facebook knows what went wrong. That's not
cloudflare's problem. We may never know what happened
at facebook probably because it's very embarrassing to
facebook.
bp
<part15sbs{at}gmail{dot}com>
On 10/5/2021 9:01 AM, Robert wrote:
I really don't like that cloudflare posted this, as it
doesn't do anything to tell why. It's not a root
cause, it's an effect and cloudflare looks like they
are just trying to say how smart they are.
On 10/5/21 8:40 AM, Bill Prince wrote:
https://blog.cloudflare.com/october-2021-facebook-outage/
bp
<part15sbs{at}gmail{dot}com>
On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
Sign him up for a SNAP card.
Sent from my iPhone
On Oct 5, 2021, at 8:25 AM, Bill Prince
mailto:part15...@gmail.com wrote:
Heard the Zuch got around $6 or $7 billion trimmed
from his net worth yesterday. Now he's only worth
around $121 billion.
bp
<part15sbs{at}gmail{dot}com>
On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
I am very curious what it was that took 6 hours to
figure out or undo.
…..but I wouldn’t judge too harshly. When any
contraption gets big and complex it’s easy to
overlook one seemingly small thing.
Remember this XKCD?
<image001.png>
*From:* AF mailto:af-boun...@af.afmug.com *On
Behalf Of *Steve Jones
*Sent:* Monday, October 04, 2021 6:51 PM
*To:* AnimalFarm Microwave Users Group
mailto:af@af.afmug.com
*Subject:* Re: [AFMUG] facebook outage
so it turns out FB murdered the announcements for
the DNS server prefixes, including the registrar.
https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
This really seems like an elementary design flaw.
Even a moron like me knows not to poke the DNS
subnets all at once, and we probably get way less
than 1k daily lookups across all the domains we
host DNS for. Ill bet nobody involved in this
makes less than 80k a year. facebook lost 4% stock
value today attributed to this on top of the
almost percent attributed to the "whistleblower".
I get nervous when I lose the company a number 10
wrench.
On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller
<par...@cyberbroadband.net> wrote:
it is alive here again
----- Original Message -----
*From:*CBB - Jay Fuller
*To:*AnimalFarm Microwave Users Group
*Sent:*Monday, October 4, 2021 4:44 PM
*Subject:*Re: [AFMUG] facebook outage
yup. resolving here. but nothing coming
up yet.
----- Original Message -----
*From:*Mike Hammett
*To:*AnimalFarm Microwave Users Group
*Sent:*Monday, October 4, 2021 4:36 PM
*Subject:*Re: [AFMUG] facebook outage
They're back... ish.
-----
Mike Hammett
Intelligent Computing Solutions
<http://www.ics-il.com/>
<https://www.facebook.com/ICSIL>
<~WRD0002.jpg>
<https://www.facebook.com/ICSIL>
<~WRD0002.jpg>
<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
<~WRD0002.jpg>
<https://www.linkedin.com/company/intelligent-computing-solutions>
<~WRD0002.jpg>
<https://twitter.com/ICSIL>
Midwest Internet Exchange
<http://www.midwest-ix.com/>
<~WRD0002.jpg>
<https://www.facebook.com/mdwestix>
<~WRD0002.jpg>
<https://www.linkedin.com/company/midwest-internet-exchange>
<~WRD0002.jpg>
<https://twitter.com/mdwestix>
The Brothers WISP
<http://www.thebrotherswisp.com/>
<~WRD0002.jpg>
<https://www.facebook.com/thebrotherswisp>
<~WRD0002.jpg>
<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
------------------------------------------------------------------------
*From: *"CBB - Jay Fuller"
<par...@cyberbroadband.net>
*To: *af@af.afmug.com
*Sent: *Monday, October 4, 2021 1:13:01 PM
*Subject: *[AFMUG] facebook outage
anyone else feeling alone and isolated
without facebook? I need my safe
space (no, not myspace) back.
babaahahah.
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
------------------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
------------------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
Carl Peterson
*PORT NETWORKS*
401 E Pratt St, Ste 2553
Baltimore, MD 21202
(410) 637-3707
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
------------------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com <mailto:AF@af.afmug.com>
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com <mailto:AF@af.afmug.com>
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
