Re: [AFMUG] facebook outage

2021-10-05 Thread Jaime Solorza 
Nope . This guy did it .

On Tue, Oct 5, 2021, 2:46 PM Nate Burke  wrote:

> I think the more plausible explanation is that Zuckerberg was in a bar
> with some guys he was trying to impress saying something like 'I do SO run
> Facebook' and then whips out his phone and instead of deleting their
> profile, he presses the 'delete all' button by accident.
>
> I've been at some small businesses where they can't change an Ethernet
> port speed without a 3 person peer review of the config update,  I have
> trouble believing that one guy mistyped something.
>
> On 10/5/2021 12:17 PM, Steve Jones wrote:
>
> ill bet whatever their process was, its not what it is and delayed
> commits/fallbacks are instituted forthwith. Not by the guy who
> committed this set of changes though, hes at the bottom of a river now
>
> On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson 
> wrote:
>
>> I wonder if they did a "commit confirmed 30”? If not, would that have
>> avoided the whole thing?
>>
>>
>> On Oct 5, 2021, at 11:08 AM, Carl Peterson 
>> wrote:
>>
>> "an engineer put in a config change on a BGP router that propagated to
>> the other BGP routers and cut off access"
>>
>> Sounds like something I would do which is why I generally insist on
>> having a remote tech physically in the building with a console cable if I
>> don't have OOB console access when changing BGP on my edge.
>>
>> On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:
>>
>>> FB has posted that an engineer put in a config change on a BGP router
>>> that propagated to the other BGP routers and cut off access.  Sounds like
>>> it took a while to track down the config change.  I do know someone who is
>>> intimate with that group but he's not talking...   Drinking, but not
>>> talking...
>>>
>>> On 10/5/21 9:17 AM, Bill Prince wrote:
>>>
>>> I see it differently. In the beginning cloudflare thought something was
>>> wrong with their systems. It doesn't explain what happened inside facebook,
>>> probably because only facebook knows what went wrong. That's not
>>> cloudflare's problem. We may never know what happened at facebook probably
>>> because it's very embarrassing to facebook.
>>>
>>>
>>> bp
>>> 
>>>
>>> On 10/5/2021 9:01 AM, Robert wrote:
>>>
>>> I really don't like that cloudflare posted this, as it doesn't do
>>> anything to tell why.  It's not a root cause, it's an effect and cloudflare
>>> looks like they are just trying to say how smart they are.
>>>
>>> On 10/5/21 8:40 AM, Bill Prince wrote:
>>>
>>>
>>> https://blog.cloudflare.com/october-2021-facebook-outage/
>>>
>>>
>>> bp
>>> 
>>>
>>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>>>
>>> Sign him up for a SNAP card.
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 5, 2021, at 8:25 AM, Bill Prince 
>>>   wrote:
>>>
>>> 
>>>
>>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
>>> yesterday. Now he's only worth around $121 billion.
>>>
>>>
>>> bp
>>> 
>>>
>>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>>>
>>> I am very curious what it was that took 6 hours to figure out or undo.
>>>
>>>
>>>
>>> …..but I wouldn’t judge too harshly.  When any contraption gets big and
>>> complex it’s easy to overlook one seemingly small thing.
>>>
>>>
>>>
>>> Remember this XKCD?
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* AF  
>>>  *On Behalf Of *Steve Jones
>>> *Sent:* Monday, October 04, 2021 6:51 PM
>>> *To:* AnimalFarm Microwave Users Group 
>>>  
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>> so it turns out FB murdered the announcements for the DNS server
>>> prefixes, including the registrar.
>>>
>>>
>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>>
>>>
>>>
>>> This really seems like an elementary design flaw. Even a moron like me
>>> knows not to poke the DNS subnets all at once, and we probably get way less
>>> than 1k daily lookups across all the domains we host DNS for. Ill bet
>>> nobody involved in this makes less than 80k a year. facebook lost 4% stock
>>> value today attributed to this on top of the almost percent attributed to
>>> the "whistleblower". I get nervous when I lose the company a number 10
>>> wrench.
>>>
>>>
>>>
>>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <
>>> par...@cyberbroadband.net> wrote:
>>>
>>>
>>>
>>> it is alive here again
>>>
>>>
>>>
>>> - Original Message -
>>>
>>> *From:* CBB - Jay Fuller 
>>>
>>> *To:* AnimalFarm Microwave Users Group 
>>>
>>> *Sent:* Monday, October 4, 2021 4:44 PM
>>>
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>>
>>>
>>> yup.  resolving here.  but nothing coming up yet.
>>>
>>>
>>>
>>> - Original Message -
>>>
>>> *From:* Mike Hammett 
>>>
>>> *To:* AnimalFarm Microwave Users Group 
>>>
>>> *Sent:* Monday, October 4, 2021 4:36 PM
>>>
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>> They're back...  ish.
>>>
>>>
>>>
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions

Re: [AFMUG] facebook outage

2021-10-05 Thread Chuck McCown via AF 
Ping FB.com -t

Sent from my iPhone

> On Oct 5, 2021, at 4:40 PM, Robert  wrote:
> 
>  Much more information,   _Except the Command_   LOL
> 
> https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
> 
>> On 10/5/21 9:17 AM, Bill Prince wrote:
>> I see it differently. In the beginning cloudflare thought something was 
>> wrong with their systems. It doesn't explain what happened inside facebook, 
>> probably because only facebook knows what went wrong. That's not 
>> cloudflare's problem. We may never know what happened at facebook probably 
>> because it's very embarrassing to facebook.
>> 
>> 
>> 
>> bp
>> 
>>> On 10/5/2021 9:01 AM, Robert wrote:
>>> I really don't like that cloudflare posted this, as it doesn't do anything 
>>> to tell why.  It's not a root cause, it's an effect and cloudflare looks 
>>> like they are just trying to say how smart they are.
>>> 
>>> On 10/5/21 8:40 AM, Bill Prince wrote:
 
 https://blog.cloudflare.com/october-2021-facebook-outage/
 
 
 
 bp
 
 On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
> Sign him up for a SNAP card.
> 
> Sent from my iPhone
> 
>> On Oct 5, 2021, at 8:25 AM, Bill Prince  wrote:
>> 
>> 
>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth 
>> yesterday. Now he's only worth around $121 billion.
>> 
>> 
>> 
>> bp
>> 
>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>>> I am very curious what it was that took 6 hours to figure out or undo.
>>>  
>>> …..but I wouldn’t judge too harshly.  When any contraption gets big and 
>>> complex it’s easy to overlook one seemingly small thing.
>>>  
>>> Remember this XKCD?
>>>  
>>> 
>>>  
>>>  
>>>  
>>> From: AF  On Behalf Of Steve Jones
>>> Sent: Monday, October 04, 2021 6:51 PM
>>> To: AnimalFarm Microwave Users Group 
>>> Subject: Re: [AFMUG] facebook outage
>>>  
>>> so it turns out FB murdered the announcements for the DNS server 
>>> prefixes, including the registrar.
>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>>  
>>> This really seems like an elementary design flaw. Even a moron like me 
>>> knows not to poke the DNS subnets all at once, and we probably get way 
>>> less than 1k daily lookups across all the domains we host DNS for. Ill 
>>> bet nobody involved in this makes less than 80k a year. facebook lost 
>>> 4% stock value today attributed to this on top of the almost percent 
>>> attributed to the "whistleblower". I get nervous when I lose the 
>>> company a number 10 wrench.
>>>  
>>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
>>>  wrote:
>>>  
>>> it is alive here again
>>>  
>>> - Original Message -
>>> From: CBB - Jay Fuller
>>> To: AnimalFarm Microwave Users Group
>>> Sent: Monday, October 4, 2021 4:44 PM
>>> Subject: Re: [AFMUG] facebook outage
>>>  
>>>  
>>> yup.  resolving here.  but nothing coming up yet.
>>>  
>>> - Original Message -
>>> From: Mike Hammett
>>> To: AnimalFarm Microwave Users Group
>>> Sent: Monday, October 4, 2021 4:36 PM
>>> Subject: Re: [AFMUG] facebook outage
>>>  
>>> They're back...  ish.
>>> 
>>> 
>>> 
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> <~WRD0002.jpg>
>>> <~WRD0002.jpg>
>>> <~WRD0002.jpg>
>>> <~WRD0002.jpg>
>>> 
>>> Midwest Internet Exchange
>>> <~WRD0002.jpg>
>>> <~WRD0002.jpg>
>>> <~WRD0002.jpg>
>>> 
>>> The Brothers WISP
>>> <~WRD0002.jpg>
>>> <~WRD0002.jpg>
>>> 
>>> 
>>> 
>>> From: "CBB - Jay Fuller" 
>>> To: af@af.afmug.com
>>> Sent: Monday, October 4, 2021 1:13:01 PM
>>> Subject: [AFMUG] facebook outage
>>> 
>>>  
>>> anyone else feeling alone and isolated without facebook?  I need my 
>>> safe space (no, not myspace) back.
>>>  
>>> babaahahah.
>>>  
>>>  
>>>  
>>> 
>>> -- 
>>> AF mailing list
>>> AF@af.afmug.com
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>>  
>>> -- 
>>> AF mailing list
>>> AF@af.afmug.com
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>> -- 
>>> AF mailing list
>>> AF@af.afmug.com
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>> -- 
>>> AF mailing list
>>> AF@af.afmug.com
>>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>> 
>>> 
>> -- 
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
> 
 
>>> 
>>> 
>> 
> 
> -- 
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- 
AF mailing list
AF@af.afmug.com
http://a

Re: [AFMUG] facebook outage

2021-10-05 Thread Robert 

Much more information,   _Except the Command_   LOL

https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

On 10/5/21 9:17 AM, Bill Prince wrote:


I see it differently. In the beginning cloudflare thought something 
was wrong with their systems. It doesn't explain what happened inside 
facebook, probably because only facebook knows what went wrong. That's 
not cloudflare's problem. We may never know what happened at facebook 
probably because it's very embarrassing to facebook.



bp

On 10/5/2021 9:01 AM, Robert wrote:
I really don't like that cloudflare posted this, as it doesn't do 
anything to tell why.  It's not a root cause, it's an effect and 
cloudflare looks like they are just trying to say how smart they are.


On 10/5/21 8:40 AM, Bill Prince wrote:



https://blog.cloudflare.com/october-2021-facebook-outage/ 




bp

On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.

Sent from my iPhone


On Oct 5, 2021, at 8:25 AM, Bill Prince  wrote:



Heard the Zuch got around $6 or $7 billion trimmed from his net 
worth yesterday. Now he's only worth around $121 billion.



bp

On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:


I am very curious what it was that took 6 hours to figure out or 
undo.


…..but I wouldn’t judge too harshly.  When any contraption gets 
big and complex it’s easy to overlook one seemingly small thing.


Remember this XKCD?



*From:* AF  *On Behalf Of *Steve Jones
*Sent:* Monday, October 04, 2021 6:51 PM
*To:* AnimalFarm Microwave Users Group 
*Subject:* Re: [AFMUG] facebook outage

so it turns out FB murdered the announcements for the DNS server 
prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ 



This really seems like an elementary design flaw. Even a moron 
like me knows not to poke the DNS subnets all at once, and we 
probably get way less than 1k daily lookups across all the 
domains we host DNS for. Ill bet nobody involved in this makes 
less than 80k a year. facebook lost 4% stock value today 
attributed to this on top of the almost percent attributed to the 
"whistleblower". I get nervous when I lose the company a number 
10 wrench.


On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
mailto:par...@cyberbroadband.net>> wrote:


it is alive here again

- Original Message -

*From:*CBB - Jay Fuller 

*To:*AnimalFarm Microwave Users Group


*Sent:*Monday, October 4, 2021 4:44 PM

*Subject:*Re: [AFMUG] facebook outage

yup. resolving here.  but nothing coming up yet.

- Original Message -

*From:*Mike Hammett 

*To:*AnimalFarm Microwave Users Group


*Sent:*Monday, October 4, 2021 4:36 PM

*Subject:*Re: [AFMUG] facebook outage

They're back...  ish.



-
Mike Hammett
Intelligent Computing Solutions 


<~WRD0002.jpg> 
<~WRD0002.jpg>

<~WRD0002.jpg>

<~WRD0002.jpg>

Midwest Internet Exchange 
<~WRD0002.jpg>

<~WRD0002.jpg>

<~WRD0002.jpg>

The Brothers WISP 
<~WRD0002.jpg>

<~WRD0002.jpg>







*From: *"CBB - Jay Fuller" mailto:par...@cyberbroadband.net>>
*To: *af@af.afmug.com 
*Sent: *Monday, October 4, 2021 1:13:01 PM
*Subject: *[AFMUG] facebook outage

anyone else feeling alone and isolated without
facebook?  I need my safe space (no, not myspace) back.

babaahahah.


-- 
AF mailing list

AF@af.afmug.com 
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] facebook outage

2021-10-05 Thread Steve Jones 
Probably better luck finding a 10mm wrench

On Tue, Oct 5, 2021, 4:26 PM Nate Burke  wrote:

> No, they just had to find a laptop that had a serial port on it.  They
> probably only are issued company tablets.
>
> On 10/5/2021 4:22 PM, Steve Jones wrote:
>
> Id like to think some lowly janitor was picked up from his vacation by a
> helicopter right from his john boat on the lake, whisked to an airport and
> loaded onto a leer jet to california, then police escorted to FB HQ to pull
> out his big ol key ring and open the door
>
> On Tue, Oct 5, 2021 at 4:12 PM Chuck McCown via AF 
> wrote:
>
>> Probably more like some kind of automation with a bug in it.  I got a
>> kick out of FB actually saying that this caused some of their people to be
>> physically locked out of data center equipment they needed to access to fix
>> it.  I don’t think I would have ever admitted that.
>>
>> *From:* Nate Burke
>> *Sent:* Tuesday, October 5, 2021 2:45 PM
>> *To:* AnimalFarm Microwave Users Group
>> *Subject:* Re: [AFMUG] facebook outage
>>
>> I think the more plausible explanation is that Zuckerberg was in a bar
>> with some guys he was trying to impress saying something like 'I do SO run
>> Facebook' and then whips out his phone and instead of deleting their
>> profile, he presses the 'delete all' button by accident.
>>
>> I've been at some small businesses where they can't change an Ethernet
>> port speed without a 3 person peer review of the config update,  I have
>> trouble believing that one guy mistyped something.
>>
>> On 10/5/2021 12:17 PM, Steve Jones wrote:
>>
>> ill bet whatever their process was, its not what it is and delayed
>> commits/fallbacks are instituted forthwith. Not by the guy who committed
>> this set of changes though, hes at the bottom of a river now
>>
>> On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson 
>> wrote:
>>
>>> I wonder if they did a "commit confirmed 30”? If not, would that have
>>> avoided the whole thing?
>>>
>>>
>>> On Oct 5, 2021, at 11:08 AM, Carl Peterson 
>>> wrote:
>>>
>>> "an engineer put in a config change on a BGP router that propagated to
>>> the other BGP routers and cut off access"
>>>
>>> Sounds like something I would do which is why I generally insist on
>>> having a remote tech physically in the building with a console cable if I
>>> don't have OOB console access when changing BGP on my edge.
>>>
>>> On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:
>>>
 FB has posted that an engineer put in a config change on a BGP router
 that propagated to the other BGP routers and cut off access.  Sounds like
 it took a while to track down the config change.  I do know someone who is
 intimate with that group but he's not talking...   Drinking, but not
 talking...

 On 10/5/21 9:17 AM, Bill Prince wrote:

 I see it differently. In the beginning cloudflare thought something was
 wrong with their systems. It doesn't explain what happened inside facebook,
 probably because only facebook knows what went wrong. That's not
 cloudflare's problem. We may never know what happened at facebook probably
 because it's very embarrassing to facebook.



 bp
 

 On 10/5/2021 9:01 AM, Robert wrote:

 I really don't like that cloudflare posted this, as it doesn't do
 anything to tell why.  It's not a root cause, it's an effect and cloudflare
 looks like they are just trying to say how smart they are.

 On 10/5/21 8:40 AM, Bill Prince wrote:



 https://blog.cloudflare.com/october-2021-facebook-outage/



 bp
 

 On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

 Sign him up for a SNAP card.

 Sent from my iPhone

 On Oct 5, 2021, at 8:25 AM, Bill Prince mailto:part15...@gmail.com
  wrote:

 

 Heard the Zuch got around $6 or $7 billion trimmed from his net worth
 yesterday. Now he's only worth around $121 billion.



 bp
 

 On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:

 I am very curious what it was that took 6 hours to figure out or undo.



 …..but I wouldn’t judge too harshly.  When any contraption gets big and
 complex it’s easy to overlook one seemingly small thing.



 Remember this XKCD?


 







 *From:* AF mailto:af-boun...@af.afmug.com  *On
 Behalf Of *Steve Jones
 *Sent:* Monday, October 04, 2021 6:51 PM
 *To:* AnimalFarm Microwave Users Group mailto:af@af.afmug.com
 
 *Subject:* Re: [AFMUG] facebook outage



 so it turns out FB murdered the announcements for the DNS server
 prefixes, including the registrar.


 https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ



 This really seems like an elementary design flaw. Even a moron like m

Re: [AFMUG] facebook outage

2021-10-05 Thread Nate Burke 
No, they just had to find a laptop that had a serial port on it. They 
probably only are issued company tablets.


On 10/5/2021 4:22 PM, Steve Jones wrote:
Id like to think some lowly janitor was picked up from his vacation by 
a helicopter right from his john boat on the lake, whisked to an 
airport and loaded onto a leer jet to california, then police escorted 
to FB HQ to pull out his big ol key ring and open the door


On Tue, Oct 5, 2021 at 4:12 PM Chuck McCown via AF > wrote:


Probably more like some kind of automation with a bug in it.  I
got a kick out of FB actually saying that this caused some of
their people to be physically locked out of data center equipment
they needed to access to fix it.  I don’t think I would have ever
admitted that.
*From:* Nate Burke
*Sent:* Tuesday, October 5, 2021 2:45 PM
*To:* AnimalFarm Microwave Users Group
*Subject:* Re: [AFMUG] facebook outage
I think the more plausible explanation is that Zuckerberg was in a
bar with some guys he was trying to impress saying something like
'I do SO run Facebook' and then whips out his phone and instead of
deleting their profile, he presses the 'delete all' button by
accident.

I've been at some small businesses where they can't change an
Ethernet port speed without a 3 person peer review of the config
update,  I have trouble believing that one guy mistyped something.

On 10/5/2021 12:17 PM, Steve Jones wrote:

ill bet whatever their process was, its not what it is and
delayed commits/fallbacks are instituted forthwith. Not by the
guy who committed this set of changes though, hes at the bottom
of a river now
On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson
 wrote:

I wonder if they did a "commit confirmed 30”? If not, would
that have avoided the whole thing?


On Oct 5, 2021, at 11:08 AM, Carl Peterson
 wrote:
"an engineer put in a config change on a BGP router that
propagated to the other BGP routers and cut off access"

Sounds like something I would do which is why I generally
insist on having a remote tech physically in the building
with a console cable if I don't have OOB console access when
changing BGP on my edge.
On Tue, Oct 5, 2021 at 11:34 AM Robert
 wrote:

FB has posted that an engineer put in a config change on
a BGP router that propagated to the other BGP routers
and cut off access.  Sounds like it took a while to
track down the config change.  I do know someone who is
intimate with that group but he's not talking...
Drinking, but not talking...

On 10/5/21 9:17 AM, Bill Prince wrote:


I see it differently. In the beginning cloudflare
thought something was wrong with their systems. It
doesn't explain what happened inside facebook, probably
because only facebook knows what went wrong. That's not
cloudflare's problem. We may never know what happened
at facebook probably because it's very embarrassing to
facebook.

bp

On 10/5/2021 9:01 AM, Robert wrote:

I really don't like that cloudflare posted this, as it
doesn't do anything to tell why. It's not a root
cause, it's an effect and cloudflare looks like they
are just trying to say how smart they are.

On 10/5/21 8:40 AM, Bill Prince wrote:


https://blog.cloudflare.com/october-2021-facebook-outage/

bp

On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.

Sent from my iPhone


On Oct 5, 2021, at 8:25 AM, Bill Prince
mailto:part15...@gmail.com wrote:



Heard the Zuch got around $6 or $7 billion trimmed
from his net worth yesterday. Now he's only worth
around $121 billion.

bp

On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:


I am very curious what it was that took 6 hours to
figure out or undo.

…..but I wouldn’t judge too harshly.  When any
contraption gets big and complex it’s easy to
overlook one seemingly small thing.

Remember this XKCD?



*From:* AF mailto:af-boun...@af.afmug.com *On
Behalf Of *Steve Jones
*Sent:* Monday, October 04, 2021 6:51 PM
*To:* AnimalFarm Microwave Users Group
mailto:af@af.afmug.com
*Subject:* Re: [AFMUG] facebook outage

so it turns out FB murdered the announcements for
the DNS server prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook

Re: [AFMUG] facebook outage

2021-10-05 Thread Steve Jones 
Id like to think some lowly janitor was picked up from his vacation by a
helicopter right from his john boat on the lake, whisked to an airport and
loaded onto a leer jet to california, then police escorted to FB HQ to pull
out his big ol key ring and open the door

On Tue, Oct 5, 2021 at 4:12 PM Chuck McCown via AF  wrote:

> Probably more like some kind of automation with a bug in it.  I got a kick
> out of FB actually saying that this caused some of their people to be
> physically locked out of data center equipment they needed to access to fix
> it.  I don’t think I would have ever admitted that.
>
> *From:* Nate Burke
> *Sent:* Tuesday, October 5, 2021 2:45 PM
> *To:* AnimalFarm Microwave Users Group
> *Subject:* Re: [AFMUG] facebook outage
>
> I think the more plausible explanation is that Zuckerberg was in a bar
> with some guys he was trying to impress saying something like 'I do SO run
> Facebook' and then whips out his phone and instead of deleting their
> profile, he presses the 'delete all' button by accident.
>
> I've been at some small businesses where they can't change an Ethernet
> port speed without a 3 person peer review of the config update,  I have
> trouble believing that one guy mistyped something.
>
> On 10/5/2021 12:17 PM, Steve Jones wrote:
>
> ill bet whatever their process was, its not what it is and delayed
> commits/fallbacks are instituted forthwith. Not by the guy who committed
> this set of changes though, hes at the bottom of a river now
>
> On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson 
> wrote:
>
>> I wonder if they did a "commit confirmed 30”? If not, would that have
>> avoided the whole thing?
>>
>>
>> On Oct 5, 2021, at 11:08 AM, Carl Peterson 
>> wrote:
>>
>> "an engineer put in a config change on a BGP router that propagated to
>> the other BGP routers and cut off access"
>>
>> Sounds like something I would do which is why I generally insist on
>> having a remote tech physically in the building with a console cable if I
>> don't have OOB console access when changing BGP on my edge.
>>
>> On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:
>>
>>> FB has posted that an engineer put in a config change on a BGP router
>>> that propagated to the other BGP routers and cut off access.  Sounds like
>>> it took a while to track down the config change.  I do know someone who is
>>> intimate with that group but he's not talking...   Drinking, but not
>>> talking...
>>>
>>> On 10/5/21 9:17 AM, Bill Prince wrote:
>>>
>>> I see it differently. In the beginning cloudflare thought something was
>>> wrong with their systems. It doesn't explain what happened inside facebook,
>>> probably because only facebook knows what went wrong. That's not
>>> cloudflare's problem. We may never know what happened at facebook probably
>>> because it's very embarrassing to facebook.
>>>
>>>
>>>
>>> bp
>>> 
>>>
>>> On 10/5/2021 9:01 AM, Robert wrote:
>>>
>>> I really don't like that cloudflare posted this, as it doesn't do
>>> anything to tell why.  It's not a root cause, it's an effect and cloudflare
>>> looks like they are just trying to say how smart they are.
>>>
>>> On 10/5/21 8:40 AM, Bill Prince wrote:
>>>
>>>
>>>
>>> https://blog.cloudflare.com/october-2021-facebook-outage/
>>>
>>>
>>>
>>> bp
>>> 
>>>
>>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>>>
>>> Sign him up for a SNAP card.
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 5, 2021, at 8:25 AM, Bill Prince mailto:part15...@gmail.com
>>> wrote:
>>>
>>> 
>>>
>>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
>>> yesterday. Now he's only worth around $121 billion.
>>>
>>>
>>>
>>> bp
>>> 
>>>
>>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>>>
>>> I am very curious what it was that took 6 hours to figure out or undo.
>>>
>>>
>>>
>>> …..but I wouldn’t judge too harshly.  When any contraption gets big and
>>> complex it’s easy to overlook one seemingly small thing.
>>>
>>>
>>>
>>> Remember this XKCD?
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* AF mailto:af-boun...@af.afmug.com *On Behalf Of *Steve Jones
>>> *Sent:* Monday, October 04, 2021 6:51 PM
>>> *To:* AnimalFarm Microwave Users Group mailto:af@af.afmug.com
>>> *Subject:* Re: [AFMUG] facebook outage
>>>
>>>
>>>
>>> so it turns out FB murdered the announcements for the DNS server
>>> prefixes, including the registrar.
>>>
>>>
>>> 
>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>>
>>>
>>>
>>> This really seems like an elementary design flaw. Even a moron like me
>>> knows not to poke the DNS subnets all at once, and we probably get way less
>>> than 1k daily lookups across all the domains we host DNS for. Ill bet
>>> nobody involved in this makes less than 80k a year. facebook lost 4% stock
>>> value today attributed to this on top of the almo

Re: [AFMUG] facebook outage

2021-10-05 Thread Chuck McCown via AF 
Probably more like some kind of automation with a bug in it.  I got a kick out 
of FB actually saying that this caused some of their people to be physically 
locked out of data center equipment they needed to access to fix it.  I don’t 
think I would have ever admitted that.  

From: Nate Burke 
Sent: Tuesday, October 5, 2021 2:45 PM
To: AnimalFarm Microwave Users Group 
Subject: Re: [AFMUG] facebook outage

I think the more plausible explanation is that Zuckerberg was in a bar with 
some guys he was trying to impress saying something like 'I do SO run Facebook' 
and then whips out his phone and instead of deleting their profile, he presses 
the 'delete all' button by accident.  

I've been at some small businesses where they can't change an Ethernet port 
speed without a 3 person peer review of the config update,  I have trouble 
believing that one guy mistyped something.  


On 10/5/2021 12:17 PM, Steve Jones wrote:

  ill bet whatever their process was, its not what it is and delayed 
commits/fallbacks are instituted forthwith. Not by the guy who committed this 
set of changes though, hes at the bottom of a river now

  On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson  wrote:

I wonder if they did a "commit confirmed 30”? If not, would that have 
avoided the whole thing? 



  On Oct 5, 2021, at 11:08 AM, Carl Peterson  
wrote:

  "an engineer put in a config change on a BGP router that propagated to 
the other BGP routers and cut off access"  


  Sounds like something I would do which is why I generally insist on 
having a remote tech physically in the building with a console cable if I don't 
have OOB console access when changing BGP on my edge.  

  On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:

FB has posted that an engineer put in a config change on a BGP router 
that propagated to the other BGP routers and cut off access.  Sounds like it 
took a while to track down the config change.  I do know someone who is 
intimate with that group but he's not talking...   Drinking, but not talking...


On 10/5/21 9:17 AM, Bill Prince wrote:

  I see it differently. In the beginning cloudflare thought something 
was wrong with their systems. It doesn't explain what happened inside facebook, 
probably because only facebook knows what went wrong. That's not cloudflare's 
problem. We may never know what happened at facebook probably because it's very 
embarrassing to facebook.




bp
On 10/5/2021 9:01 AM, Robert wrote:

I really don't like that cloudflare posted this, as it doesn't do 
anything to tell why.  It's not a root cause, it's an effect and cloudflare 
looks like they are just trying to say how smart they are.


On 10/5/21 8:40 AM, Bill Prince wrote:



  https://blog.cloudflare.com/october-2021-facebook-outage/



bp
On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.


Sent from my iPhone


  On Oct 5, 2021, at 8:25 AM, Bill Prince 
mailto:part15...@gmail.com wrote:


   
  Heard the Zuch got around $6 or $7 billion trimmed from his 
net worth yesterday. Now he's only worth around $121 billion.



bp
On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:

I am very curious what it was that took 6 hours to figure 
out or undo.



…..but I wouldn’t judge too harshly.  When any contraption 
gets big and complex it’s easy to overlook one seemingly small thing.



Remember this XKCD?










From: AF mailto:af-boun...@af.afmug.com On Behalf Of Steve 
Jones
Sent: Monday, October 04, 2021 6:51 PM
To: AnimalFarm Microwave Users Group mailto:af@af.afmug.com
Subject: Re: [AFMUG] facebook outage



so it turns out FB murdered the announcements for the DNS 
server prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ



This really seems like an elementary design flaw. Even a 
moron like me knows not to poke the DNS subnets all at once, and we probably 
get way less than 1k daily lookups across all the domains we host DNS for. Ill 
bet nobody involved in this makes less than 80k a year. facebook lost 4% stock 
value today attributed to this on top of the almost percent attributed to the 
"whistleblower". I get nervous when I lose the company a number 10 wrench.



On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
 wrote:



  it is alive here again



- Original Message - 

From: CBB - Jay Fuller 

To: AnimalFarm Microwave Users Group 

Sent: Monday, October 4, 2021

Re: [AFMUG] facebook outage

2021-10-05 Thread Nate Burke 
I think the more plausible explanation is that Zuckerberg was in a bar 
with some guys he was trying to impress saying something like 'I do SO 
run Facebook' and then whips out his phone and instead of deleting their 
profile, he presses the 'delete all' button by accident.


I've been at some small businesses where they can't change an Ethernet 
port speed without a 3 person peer review of the config update,  I have 
trouble believing that one guy mistyped something.


On 10/5/2021 12:17 PM, Steve Jones wrote:
ill bet whatever their process was, its not what it is and delayed 
commits/fallbacks are instituted forthwith. Not by the guy who 
committed this set of changes though, hes at the bottom of a river now


On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson > wrote:


I wonder if they did a "commit confirmed 30”? If not, would that
have avoided the whole thing?



On Oct 5, 2021, at 11:08 AM, Carl Peterson
mailto:cpeter...@portnetworks.com>>
wrote:

"an engineer put in a config change on a BGP router that
propagated to the other BGP routers and cut off access"

Sounds like something I would do which is why I generally insist
on having a remote tech physically in the building with a console
cable if I don't have OOB console access when changing BGP on my
edge.

On Tue, Oct 5, 2021 at 11:34 AM Robert mailto:i...@avantwireless.com>> wrote:

FB has posted that an engineer put in a config change on a
BGP router that propagated to the other BGP routers and cut
off access.  Sounds like it took a while to track down the
config change.  I do know someone who is intimate with that
group but he's not talking... Drinking, but not talking...

On 10/5/21 9:17 AM, Bill Prince wrote:


I see it differently. In the beginning cloudflare thought
something was wrong with their systems. It doesn't explain
what happened inside facebook, probably because only
facebook knows what went wrong. That's not cloudflare's
problem. We may never know what happened at facebook
probably because it's very embarrassing to facebook.


bp

On 10/5/2021 9:01 AM, Robert wrote:

I really don't like that cloudflare posted this, as it
doesn't do anything to tell why.  It's not a root cause,
it's an effect and cloudflare looks like they are just
trying to say how smart they are.

On 10/5/21 8:40 AM, Bill Prince wrote:



https://blog.cloudflare.com/october-2021-facebook-outage/


bp

On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.

Sent from my iPhone


On Oct 5, 2021, at 8:25 AM, Bill Prince
  wrote:



Heard the Zuch got around $6 or $7 billion trimmed from
his net worth yesterday. Now he's only worth around $121
billion.


bp

On 10/5/2021 6:52 AM, dmmoff...@gmail.com
 wrote:


I am very curious what it was that took 6 hours to
figure out or undo.

…..but I wouldn’t judge too harshly. When any
contraption gets big and complex it’s easy to overlook
one seemingly small thing.

Remember this XKCD?



*From:* AF 
 *On Behalf Of *Steve Jones
*Sent:* Monday, October 04, 2021 6:51 PM
*To:* AnimalFarm Microwave Users Group
 
*Subject:* Re: [AFMUG] facebook outage

so it turns out FB murdered the announcements for the
DNS server prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ

This really seems like an elementary design flaw. Even
a moron like me knows not to poke the DNS subnets all
at once, and we probably get way less than 1k daily
lookups across all the domains we host DNS for. Ill bet
nobody involved in this makes less than 80k a year.
facebook lost 4% stock value today attributed to this
on top of the almost percent attributed to the
"whistleblower". I get nervous when I lose the company
a number 10 wrench.

On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller
mailto:par...@cyberbroadband.net>> wrote:

it is alive here again

- Original Message -

*From:*CBB - Jay Fuller


*To:*AnimalFarm Microwave Users Group


*Sent:*Monday, October 4, 2021 4:44 PM

*Subject:*Re: [AFMUG] facebook outage

yup.  resolving here.  but nothing com

Re: [AFMUG] facebook outage

2021-10-05 Thread Chuck McCown via AF 
If 30 minutes would have been long enough.  I wonder how long it really takes 
for BGP changes to propagate.

Having contracted for FB before, let me tell you their management style is 
horrible.  Conference calls to plan conference calls.  I got my ass chewed for 
being honest on a call.  Can’t cause shame or embarrassment.  They had an 
engineering company that royally screwed things up but I had to act like they 
were the best company on the planet during calls.  Lots of infighting between 
different FB departments.  Lots of silos.  The managers are imperious.  

In any event, I am betting they are in the process of convening a NASA type 
challenger explosion panel to have a year’s worth of conference calls to make 
even more bureaucracy to prevent something like this from happening again.  

Back in olden times, when everyone that owned a router trusted everyone else, 
someone announced they could take all IPs over BGP.  I think they were in 
Champaign Urbana.  I was in Quincy Ill at the time.  In any event it brought 
our part of the internet to a screeching halt.  I think there was an RFCs 
created that very day and solutions created in just a few more days over that 
one.  



From: Cassidy B. Larson 
Sent: Tuesday, October 5, 2021 11:12 AM
To: AnimalFarm Microwave Users Group 
Subject: Re: [AFMUG] facebook outage

I wonder if they did a "commit confirmed 30”? If not, would that have avoided 
the whole thing? 



  On Oct 5, 2021, at 11:08 AM, Carl Peterson  wrote:

  "an engineer put in a config change on a BGP router that propagated to the 
other BGP routers and cut off access"  


  Sounds like something I would do which is why I generally insist on having a 
remote tech physically in the building with a console cable if I don't have OOB 
console access when changing BGP on my edge.  

  On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:

FB has posted that an engineer put in a config change on a BGP router that 
propagated to the other BGP routers and cut off access.  Sounds like it took a 
while to track down the config change.  I do know someone who is intimate with 
that group but he's not talking...   Drinking, but not talking...


On 10/5/21 9:17 AM, Bill Prince wrote:

  I see it differently. In the beginning cloudflare thought something was 
wrong with their systems. It doesn't explain what happened inside facebook, 
probably because only facebook knows what went wrong. That's not cloudflare's 
problem. We may never know what happened at facebook probably because it's very 
embarrassing to facebook.




bp
On 10/5/2021 9:01 AM, Robert wrote:

I really don't like that cloudflare posted this, as it doesn't do 
anything to tell why.  It's not a root cause, it's an effect and cloudflare 
looks like they are just trying to say how smart they are.


On 10/5/21 8:40 AM, Bill Prince wrote:



  https://blog.cloudflare.com/october-2021-facebook-outage/



bp
On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.


Sent from my iPhone


  On Oct 5, 2021, at 8:25 AM, Bill Prince 
mailto:part15...@gmail.com wrote:


   
  Heard the Zuch got around $6 or $7 billion trimmed from his net 
worth yesterday. Now he's only worth around $121 billion.



bp
On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:

I am very curious what it was that took 6 hours to figure out 
or undo.



…..but I wouldn’t judge too harshly.  When any contraption gets 
big and complex it’s easy to overlook one seemingly small thing.



Remember this XKCD?










From: AF mailto:af-boun...@af.afmug.com On Behalf Of Steve Jones
Sent: Monday, October 04, 2021 6:51 PM
To: AnimalFarm Microwave Users Group mailto:af@af.afmug.com
Subject: Re: [AFMUG] facebook outage



so it turns out FB murdered the announcements for the DNS 
server prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ



This really seems like an elementary design flaw. Even a moron 
like me knows not to poke the DNS subnets all at once, and we probably get way 
less than 1k daily lookups across all the domains we host DNS for. Ill bet 
nobody involved in this makes less than 80k a year. facebook lost 4% stock 
value today attributed to this on top of the almost percent attributed to the 
"whistleblower". I get nervous when I lose the company a number 10 wrench.



On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
 wrote:



  it is alive here again



- Original Message - 

From: CBB - Jay Fuller 

To: AnimalFarm Microwave Users Group 

Sent: Monday, Octobe

Re: [AFMUG] facebook outage

2021-10-05 Thread Steve Jones 
ill bet whatever their process was, its not what it is and delayed
commits/fallbacks are instituted forthwith. Not by the guy who
committed this set of changes though, hes at the bottom of a river now

On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson  wrote:

> I wonder if they did a "commit confirmed 30”? If not, would that have
> avoided the whole thing?
>
>
> On Oct 5, 2021, at 11:08 AM, Carl Peterson 
> wrote:
>
> "an engineer put in a config change on a BGP router that propagated to
> the other BGP routers and cut off access"
>
> Sounds like something I would do which is why I generally insist on
> having a remote tech physically in the building with a console cable if I
> don't have OOB console access when changing BGP on my edge.
>
> On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:
>
>> FB has posted that an engineer put in a config change on a BGP router
>> that propagated to the other BGP routers and cut off access.  Sounds like
>> it took a while to track down the config change.  I do know someone who is
>> intimate with that group but he's not talking...   Drinking, but not
>> talking...
>>
>> On 10/5/21 9:17 AM, Bill Prince wrote:
>>
>> I see it differently. In the beginning cloudflare thought something was
>> wrong with their systems. It doesn't explain what happened inside facebook,
>> probably because only facebook knows what went wrong. That's not
>> cloudflare's problem. We may never know what happened at facebook probably
>> because it's very embarrassing to facebook.
>>
>>
>> bp
>> 
>>
>> On 10/5/2021 9:01 AM, Robert wrote:
>>
>> I really don't like that cloudflare posted this, as it doesn't do
>> anything to tell why.  It's not a root cause, it's an effect and cloudflare
>> looks like they are just trying to say how smart they are.
>>
>> On 10/5/21 8:40 AM, Bill Prince wrote:
>>
>>
>> https://blog.cloudflare.com/october-2021-facebook-outage/
>>
>>
>> bp
>> 
>>
>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>>
>> Sign him up for a SNAP card.
>>
>> Sent from my iPhone
>>
>> On Oct 5, 2021, at 8:25 AM, Bill Prince 
>>  wrote:
>>
>> 
>>
>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
>> yesterday. Now he's only worth around $121 billion.
>>
>>
>> bp
>> 
>>
>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>>
>> I am very curious what it was that took 6 hours to figure out or undo.
>>
>>
>>
>> …..but I wouldn’t judge too harshly.  When any contraption gets big and
>> complex it’s easy to overlook one seemingly small thing.
>>
>>
>>
>> Remember this XKCD?
>>
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> *From:* AF   *On
>> Behalf Of *Steve Jones
>> *Sent:* Monday, October 04, 2021 6:51 PM
>> *To:* AnimalFarm Microwave Users Group 
>> 
>> *Subject:* Re: [AFMUG] facebook outage
>>
>>
>>
>> so it turns out FB murdered the announcements for the DNS server
>> prefixes, including the registrar.
>>
>>
>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>
>>
>>
>> This really seems like an elementary design flaw. Even a moron like me
>> knows not to poke the DNS subnets all at once, and we probably get way less
>> than 1k daily lookups across all the domains we host DNS for. Ill bet
>> nobody involved in this makes less than 80k a year. facebook lost 4% stock
>> value today attributed to this on top of the almost percent attributed to
>> the "whistleblower". I get nervous when I lose the company a number 10
>> wrench.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <
>> par...@cyberbroadband.net> wrote:
>>
>>
>>
>> it is alive here again
>>
>>
>>
>> - Original Message -
>>
>> *From:* CBB - Jay Fuller 
>>
>> *To:* AnimalFarm Microwave Users Group 
>>
>> *Sent:* Monday, October 4, 2021 4:44 PM
>>
>> *Subject:* Re: [AFMUG] facebook outage
>>
>>
>>
>>
>>
>> yup.  resolving here.  but nothing coming up yet.
>>
>>
>>
>> - Original Message -
>>
>> *From:* Mike Hammett 
>>
>> *To:* AnimalFarm Microwave Users Group 
>>
>> *Sent:* Monday, October 4, 2021 4:36 PM
>>
>> *Subject:* Re: [AFMUG] facebook outage
>>
>>
>>
>> They're back...  ish.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions 
>> 
>> <~WRD0002.jpg> 
>> <~WRD0002.jpg>
>> 
>> <~WRD0002.jpg>
>> 
>> <~WRD0002.jpg>
>> 
>> Midwest Internet Exchange 
>> <~WRD0002.jpg>
>> 
>> <~WRD0002.jpg>
>> 
>> <~WRD0002.jpg>
>> 
>> The Brothers WISP 
>> <~WRD0002.jpg>
>> 
>> <~WRD0002.jpg>
>>
>>
>>
>> 
>> ---

Re: [AFMUG] facebook outage

2021-10-05 Thread Cassidy B. Larson 
I wonder if they did a "commit confirmed 30”? If not, would that have avoided 
the whole thing?


> On Oct 5, 2021, at 11:08 AM, Carl Peterson  wrote:
> 
> "an engineer put in a config change on a BGP router that propagated to the 
> other BGP routers and cut off access"  
> 
> Sounds like something I would do which is why I generally insist on having a 
> remote tech physically in the building with a console cable if I don't have 
> OOB console access when changing BGP on my edge.  
> 
> On Tue, Oct 5, 2021 at 11:34 AM Robert  > wrote:
> FB has posted that an engineer put in a config change on a BGP router that 
> propagated to the other BGP routers and cut off access.  Sounds like it took 
> a while to track down the config change.  I do know someone who is intimate 
> with that group but he's not talking...   Drinking, but not talking...
> 
> On 10/5/21 9:17 AM, Bill Prince wrote:
>> I see it differently. In the beginning cloudflare thought something was 
>> wrong with their systems. It doesn't explain what happened inside facebook, 
>> probably because only facebook knows what went wrong. That's not 
>> cloudflare's problem. We may never know what happened at facebook probably 
>> because it's very embarrassing to facebook.
>> 
>> 
>> 
>> bp
>> 
>> On 10/5/2021 9:01 AM, Robert wrote:
>>> I really don't like that cloudflare posted this, as it doesn't do anything 
>>> to tell why.  It's not a root cause, it's an effect and cloudflare looks 
>>> like they are just trying to say how smart they are.
>>> 
>>> On 10/5/21 8:40 AM, Bill Prince wrote:
 
 https://blog.cloudflare.com/october-2021-facebook-outage/ 
 
 
 bp
 
 On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
> Sign him up for a SNAP card.
> 
> Sent from my iPhone
> 
>> On Oct 5, 2021, at 8:25 AM, Bill Prince  
>>  wrote:
>> 
>> 
>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth 
>> yesterday. Now he's only worth around $121 billion.
>> 
>> 
>> 
>> bp
>> 
>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com  
>> wrote:
>>> I am very curious what it was that took 6 hours to figure out or undo.
>>> 
>>>  
>>> 
>>> …..but I wouldn’t judge too harshly.  When any contraption gets big and 
>>> complex it’s easy to overlook one seemingly small thing.
>>> 
>>>  
>>> 
>>> Remember this XKCD?
>>> 
>>>  
>>> 
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>> From: AF   On 
>>> Behalf Of Steve Jones
>>> Sent: Monday, October 04, 2021 6:51 PM
>>> To: AnimalFarm Microwave Users Group  
>>> 
>>> Subject: Re: [AFMUG] facebook outage
>>> 
>>>  
>>> 
>>> so it turns out FB murdered the announcements for the DNS server 
>>> prefixes, including the registrar.
>>> 
>>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>>  
>>> 
>>>  
>>> 
>>> This really seems like an elementary design flaw. Even a moron like me 
>>> knows not to poke the DNS subnets all at once, and we probably get way 
>>> less than 1k daily lookups across all the domains we host DNS for. Ill 
>>> bet nobody involved in this makes less than 80k a year. facebook lost 
>>> 4% stock value today attributed to this on top of the almost percent 
>>> attributed to the "whistleblower". I get nervous when I lose the 
>>> company a number 10 wrench.
>>> 
>>>  
>>> 
>>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
>>> mailto:par...@cyberbroadband.net>> wrote:
>>> 
>>>  
>>> 
>>> it is alive here again
>>> 
>>>  
>>> 
>>> - Original Message -
>>> 
>>> From: CBB - Jay Fuller 
>>> To: AnimalFarm Microwave Users Group 
>>> Sent: Monday, October 4, 2021 4:44 PM
>>> 
>>> Subject: Re: [AFMUG] facebook outage
>>> 
>>>  
>>> 
>>>  
>>> 
>>> yup.  resolving here.  but nothing coming up yet.
>>> 
>>>  
>>> 
>>> - Original Message -
>>> 
>>> From: Mike Hammett 
>>> To: AnimalFarm Microwave Users Group 
>>> Sent: Monday, October 4, 2021 4:36 PM
>>> 
>>> Subject: Re: [AFMUG] facebook outage
>>> 
>>>  
>>> 
>>> They're back...  ish.
>>> 
>>> 
>>> 
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions 
>>>

Re: [AFMUG] facebook outage

2021-10-05 Thread Carl Peterson 
"an engineer put in a config change on a BGP router that propagated to the
other BGP routers and cut off access"

Sounds like something I would do which is why I generally insist on
having a remote tech physically in the building with a console cable if I
don't have OOB console access when changing BGP on my edge.

On Tue, Oct 5, 2021 at 11:34 AM Robert  wrote:

> FB has posted that an engineer put in a config change on a BGP router that
> propagated to the other BGP routers and cut off access.  Sounds like it
> took a while to track down the config change.  I do know someone who is
> intimate with that group but he's not talking...   Drinking, but not
> talking...
>
> On 10/5/21 9:17 AM, Bill Prince wrote:
>
> I see it differently. In the beginning cloudflare thought something was
> wrong with their systems. It doesn't explain what happened inside facebook,
> probably because only facebook knows what went wrong. That's not
> cloudflare's problem. We may never know what happened at facebook probably
> because it's very embarrassing to facebook.
>
>
> bp
> 
>
> On 10/5/2021 9:01 AM, Robert wrote:
>
> I really don't like that cloudflare posted this, as it doesn't do anything
> to tell why.  It's not a root cause, it's an effect and cloudflare looks
> like they are just trying to say how smart they are.
>
> On 10/5/21 8:40 AM, Bill Prince wrote:
>
>
> https://blog.cloudflare.com/october-2021-facebook-outage/
>
>
> bp
> 
>
> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>
> Sign him up for a SNAP card.
>
> Sent from my iPhone
>
> On Oct 5, 2021, at 8:25 AM, Bill Prince 
>  wrote:
>
> 
>
> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
> yesterday. Now he's only worth around $121 billion.
>
>
> bp
> 
>
> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>
> I am very curious what it was that took 6 hours to figure out or undo.
>
>
>
> …..but I wouldn’t judge too harshly.  When any contraption gets big and
> complex it’s easy to overlook one seemingly small thing.
>
>
>
> Remember this XKCD?
>
>
> 
>
>
>
>
>
>
>
> *From:* AF   *On Behalf
> Of *Steve Jones
> *Sent:* Monday, October 04, 2021 6:51 PM
> *To:* AnimalFarm Microwave Users Group  
> *Subject:* Re: [AFMUG] facebook outage
>
>
>
> so it turns out FB murdered the announcements for the DNS server prefixes,
> including the registrar.
>
>
> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>
>
>
> This really seems like an elementary design flaw. Even a moron like me
> knows not to poke the DNS subnets all at once, and we probably get way less
> than 1k daily lookups across all the domains we host DNS for. Ill bet
> nobody involved in this makes less than 80k a year. facebook lost 4% stock
> value today attributed to this on top of the almost percent attributed to
> the "whistleblower". I get nervous when I lose the company a number 10
> wrench.
>
>
>
> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
> wrote:
>
>
>
> it is alive here again
>
>
>
> - Original Message -
>
> *From:* CBB - Jay Fuller 
>
> *To:* AnimalFarm Microwave Users Group 
>
> *Sent:* Monday, October 4, 2021 4:44 PM
>
> *Subject:* Re: [AFMUG] facebook outage
>
>
>
>
>
> yup.  resolving here.  but nothing coming up yet.
>
>
>
> - Original Message -
>
> *From:* Mike Hammett 
>
> *To:* AnimalFarm Microwave Users Group 
>
> *Sent:* Monday, October 4, 2021 4:36 PM
>
> *Subject:* Re: [AFMUG] facebook outage
>
>
>
> They're back...  ish.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> <~WRD0002.jpg> 
> <~WRD0002.jpg>
> 
> <~WRD0002.jpg>
> 
> <~WRD0002.jpg>
> 
> Midwest Internet Exchange 
> <~WRD0002.jpg>
> 
> <~WRD0002.jpg>
> 
> <~WRD0002.jpg>
> 
> The Brothers WISP 
> <~WRD0002.jpg>
> 
> <~WRD0002.jpg>
>
>
>
> 
> --
>
> *From: *"CBB - Jay Fuller" 
> *To: *af@af.afmug.com
> *Sent: *Monday, October 4, 2021 1:13:01 PM
> *Subject: *[AFMUG] facebook outage
>
>
>
> anyone else feeling alone and isolated without facebook?  I need my safe
> space (no, not myspace) back.
>
>
>
> babaahahah.
>
>
>
>
>
>
>
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
>
> --
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
> --
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.

Re: [AFMUG] facebook outage

2021-10-05 Thread Robert 
FB has posted that an engineer put in a config change on a BGP router 
that propagated to the other BGP routers and cut off access.  Sounds 
like it took a while to track down the config change.  I do know someone 
who is intimate with that group but he's not talking...   Drinking, but 
not talking...


On 10/5/21 9:17 AM, Bill Prince wrote:


I see it differently. In the beginning cloudflare thought something 
was wrong with their systems. It doesn't explain what happened inside 
facebook, probably because only facebook knows what went wrong. That's 
not cloudflare's problem. We may never know what happened at facebook 
probably because it's very embarrassing to facebook.



bp

On 10/5/2021 9:01 AM, Robert wrote:
I really don't like that cloudflare posted this, as it doesn't do 
anything to tell why.  It's not a root cause, it's an effect and 
cloudflare looks like they are just trying to say how smart they are.


On 10/5/21 8:40 AM, Bill Prince wrote:



https://blog.cloudflare.com/october-2021-facebook-outage/ 




bp

On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.

Sent from my iPhone


On Oct 5, 2021, at 8:25 AM, Bill Prince  wrote:



Heard the Zuch got around $6 or $7 billion trimmed from his net 
worth yesterday. Now he's only worth around $121 billion.



bp

On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:


I am very curious what it was that took 6 hours to figure out or 
undo.


…..but I wouldn’t judge too harshly.  When any contraption gets 
big and complex it’s easy to overlook one seemingly small thing.


Remember this XKCD?



*From:* AF  *On Behalf Of *Steve Jones
*Sent:* Monday, October 04, 2021 6:51 PM
*To:* AnimalFarm Microwave Users Group 
*Subject:* Re: [AFMUG] facebook outage

so it turns out FB murdered the announcements for the DNS server 
prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ 



This really seems like an elementary design flaw. Even a moron 
like me knows not to poke the DNS subnets all at once, and we 
probably get way less than 1k daily lookups across all the 
domains we host DNS for. Ill bet nobody involved in this makes 
less than 80k a year. facebook lost 4% stock value today 
attributed to this on top of the almost percent attributed to the 
"whistleblower". I get nervous when I lose the company a number 
10 wrench.


On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
mailto:par...@cyberbroadband.net>> wrote:


it is alive here again

- Original Message -

*From:*CBB - Jay Fuller 

*To:*AnimalFarm Microwave Users Group


*Sent:*Monday, October 4, 2021 4:44 PM

*Subject:*Re: [AFMUG] facebook outage

yup. resolving here.  but nothing coming up yet.

- Original Message -

*From:*Mike Hammett 

*To:*AnimalFarm Microwave Users Group


*Sent:*Monday, October 4, 2021 4:36 PM

*Subject:*Re: [AFMUG] facebook outage

They're back...  ish.



-
Mike Hammett
Intelligent Computing Solutions 


<~WRD0002.jpg> 
<~WRD0002.jpg>

<~WRD0002.jpg>

<~WRD0002.jpg>

Midwest Internet Exchange 
<~WRD0002.jpg>

<~WRD0002.jpg>

<~WRD0002.jpg>

The Brothers WISP 
<~WRD0002.jpg>

<~WRD0002.jpg>







*From: *"CBB - Jay Fuller" mailto:par...@cyberbroadband.net>>
*To: *af@af.afmug.com 
*Sent: *Monday, October 4, 2021 1:13:01 PM
*Subject: *[AFMUG] facebook outage

anyone else feeling alone and isolated without
facebook?  I need my safe space (no, not myspace) back.

babaahahah.


-- 
AF mailing list

Re: [AFMUG] facebook outage

2021-10-05 Thread Bill Prince 

  
  
I see it differently. In the beginning cloudflare thought
  something was wrong with their systems. It doesn't explain what
  happened inside facebook, probably because only facebook knows
  what went wrong. That's not cloudflare's problem. We may never
  know what happened at facebook probably because it's very
  embarrassing to facebook.



bp

On 10/5/2021 9:01 AM, Robert wrote:


  
  I really don't like that cloudflare posted
this, as it doesn't do anything to tell why.  It's not a root
cause, it's an effect and cloudflare looks like they are just
trying to say how smart they are.
  
  On 10/5/21 8:40 AM, Bill Prince
wrote:
  
  



https://blog.cloudflare.com/october-2021-facebook-outage/


bp

On 10/5/2021 7:39 AM, Chuck McCown
  via AF wrote:


  
  Sign him up for a SNAP card.
  
  Sent from my iPhone
  
On Oct 5, 2021, at 8:25 AM, Bill
  Prince 
  wrote:
  

  
  

  
  Heard the Zuch got around $6 or $7 billion trimmed from
his net worth yesterday. Now he's only worth around $121
billion.
  
  
  bp

  On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
  
  





  I am very curious what it was
that took 6 hours to figure out or undo.
   
  …..but I wouldn’t judge too
harshly.  When any contraption gets big and complex
it’s easy to overlook one seemingly small thing.
   
  Remember this XKCD?
   
  
  
   
   
   
  
From: AF 
  On Behalf Of Steve Jones
  Sent: Monday, October 04, 2021 6:51 PM
  To: AnimalFarm Microwave Users Group 
  Subject: Re: [AFMUG] facebook outage
  
   
  
so it turns out FB murdered the
  announcements for the DNS server prefixes,
  including the registrar.

  https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ


   


  This really seems like an
elementary design flaw. Even a moron like me
knows not to poke the DNS subnets all at once,
and we probably get way less than 1k daily
lookups across all the domains we host DNS for.
Ill bet nobody involved in this makes less than
80k a year. facebook lost 4% stock value today
attributed to this on top of the almost percent
attributed to the "whistleblower". I get nervous
when I lose the company a number 10 wrench.

  
   
  

  On Mon, Oct 4, 2021 at 4:51
PM CBB - Jay Fuller 
wrote:


  

   


  it
  is alive here again


   


  
-
Original Message - 
  
  
From:
CBB - Jay
  Fuller 
  
  
To: AnimalFarm
  Microwave Users Group 
  
  
Sent:
Monday, October 4, 2021 4:44 PM
  
  
Subject: Re:
[AFMUG] facebook outage

Re: [AFMUG] facebook outage

2021-10-05 Thread Robert 
I really don't like that cloudflare posted this, as it doesn't do 
anything to tell why.  It's not a root cause, it's an effect and 
cloudflare looks like they are just trying to say how smart they are.


On 10/5/21 8:40 AM, Bill Prince wrote:



https://blog.cloudflare.com/october-2021-facebook-outage/ 




bp

On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:

Sign him up for a SNAP card.

Sent from my iPhone


On Oct 5, 2021, at 8:25 AM, Bill Prince  wrote:



Heard the Zuch got around $6 or $7 billion trimmed from his net 
worth yesterday. Now he's only worth around $121 billion.



bp

On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:


I am very curious what it was that took 6 hours to figure out or undo.

…..but I wouldn’t judge too harshly.  When any contraption gets big 
and complex it’s easy to overlook one seemingly small thing.


Remember this XKCD?



*From:* AF  *On Behalf Of *Steve Jones
*Sent:* Monday, October 04, 2021 6:51 PM
*To:* AnimalFarm Microwave Users Group 
*Subject:* Re: [AFMUG] facebook outage

so it turns out FB murdered the announcements for the DNS server 
prefixes, including the registrar.


https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ 



This really seems like an elementary design flaw. Even a moron like 
me knows not to poke the DNS subnets all at once, and we probably 
get way less than 1k daily lookups across all the domains we host 
DNS for. Ill bet nobody involved in this makes less than 80k a 
year. facebook lost 4% stock value today attributed to this on top 
of the almost percent attributed to the "whistleblower". I get 
nervous when I lose the company a number 10 wrench.


On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller 
mailto:par...@cyberbroadband.net>> wrote:


it is alive here again

- Original Message -

*From:*CBB - Jay Fuller 

*To:*AnimalFarm Microwave Users Group 

*Sent:*Monday, October 4, 2021 4:44 PM

*Subject:*Re: [AFMUG] facebook outage

yup. resolving here.  but nothing coming up yet.

- Original Message -

*From:*Mike Hammett 

*To:*AnimalFarm Microwave Users Group


*Sent:*Monday, October 4, 2021 4:36 PM

*Subject:*Re: [AFMUG] facebook outage

They're back...  ish.



-
Mike Hammett
Intelligent Computing Solutions 


<~WRD0002.jpg> 
<~WRD0002.jpg>

<~WRD0002.jpg>

<~WRD0002.jpg>

Midwest Internet Exchange 
<~WRD0002.jpg>

<~WRD0002.jpg>

<~WRD0002.jpg>

The Brothers WISP 
<~WRD0002.jpg>

<~WRD0002.jpg>







*From: *"CBB - Jay Fuller" mailto:par...@cyberbroadband.net>>
*To: *af@af.afmug.com 
*Sent: *Monday, October 4, 2021 1:13:01 PM
*Subject: *[AFMUG] facebook outage

anyone else feeling alone and isolated without
facebook?  I need my safe space (no, not myspace) back.

babaahahah.


-- 
AF mailing list

AF@af.afmug.com 
http://af.afmug.com/mailman/listinfo/af_af.afmug.com





-- 
AF mailing list

AF@af.afmug.com 
http://af.afmug.com/mailman/listinfo/af_af.afmug.com




-- 
AF mailing list

AF@af.afmug.com 
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] facebook outage

2021-10-05 Thread Bill Prince 

  
  


https://blog.cloudflare.com/october-2021-facebook-outage/


bp

On 10/5/2021 7:39 AM, Chuck McCown via
  AF wrote:


  
  Sign him up for a SNAP card.
  
  Sent from my iPhone
  
On Oct 5, 2021, at 8:25 AM, Bill Prince
   wrote:
  

  
  

  
  Heard the Zuch got around $6 or $7 billion trimmed from his
net worth yesterday. Now he's only worth around $121
billion.
  
  
  bp

  On 10/5/2021 6:52 AM, dmmoff...@gmail.com
wrote:
  
  





  I am very curious what it was that
took 6 hours to figure out or undo.
   
  …..but I wouldn’t judge too harshly. 
When any contraption gets big and complex it’s easy to
overlook one seemingly small thing.
   
  Remember this XKCD?
   
  
  
   
   
   
  
From: AF 
  On Behalf Of Steve Jones
  Sent: Monday, October 04, 2021 6:51 PM
  To: AnimalFarm Microwave Users Group 
  Subject: Re: [AFMUG] facebook outage
  
   
  
so it turns out FB murdered the
  announcements for the DNS server prefixes, including
  the registrar.

  https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ


   


  This really seems like an
elementary design flaw. Even a moron like me knows
not to poke the DNS subnets all at once, and we
probably get way less than 1k daily lookups across
all the domains we host DNS for. Ill bet nobody
involved in this makes less than 80k a year.
facebook lost 4% stock value today attributed to
this on top of the almost percent attributed to the
"whistleblower". I get nervous when I lose the
company a number 10 wrench.

  
   
  

  On Mon, Oct 4, 2021 at 4:51 PM
CBB - Jay Fuller 
wrote:


  

   


  it
  is alive here again


   


  
-
Original Message - 
  
  
From:
CBB - Jay Fuller
  
  
  
To: AnimalFarm
  Microwave Users Group 
  
  
Sent:
Monday, October 4, 2021 4:44 PM
  
  
Subject: Re:
[AFMUG] facebook outage
  
  
 
  
  
 
  
  
yup. 
resolving here.  but nothing coming up yet.
  
  
 
  
  

  -
  Original Message - 


  From:
  Mike Hammett



  To: AnimalFarm
Microwave Users Group 


  Sent:
  Monday, October 4, 2021 4:36 PM


  Subject: Re:
  [AFMUG] facebook outage


   


  They're

Re: [AFMUG] facebook outage

2021-10-05 Thread Chuck McCown via AF 
Sign him up for a SNAP card.

Sent from my iPhone

> On Oct 5, 2021, at 8:25 AM, Bill Prince  wrote:
> 
> 
> Heard the Zuch got around $6 or $7 billion trimmed from his net worth 
> yesterday. Now he's only worth around $121 billion.
> 
> 
> 
> bp
> 
> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>> I am very curious what it was that took 6 hours to figure out or undo.
>>  
>> …..but I wouldn’t judge too harshly.  When any contraption gets big and 
>> complex it’s easy to overlook one seemingly small thing.
>>  
>> Remember this XKCD?
>>  
>> 
>>  
>>  
>>  
>> From: AF  On Behalf Of Steve Jones
>> Sent: Monday, October 04, 2021 6:51 PM
>> To: AnimalFarm Microwave Users Group 
>> Subject: Re: [AFMUG] facebook outage
>>  
>> so it turns out FB murdered the announcements for the DNS server prefixes, 
>> including the registrar.
>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>  
>> This really seems like an elementary design flaw. Even a moron like me knows 
>> not to poke the DNS subnets all at once, and we probably get way less than 
>> 1k daily lookups across all the domains we host DNS for. Ill bet nobody 
>> involved in this makes less than 80k a year. facebook lost 4% stock value 
>> today attributed to this on top of the almost percent attributed to the 
>> "whistleblower". I get nervous when I lose the company a number 10 wrench.
>>  
>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller  
>> wrote:
>>  
>> it is alive here again
>>  
>> - Original Message -
>> From: CBB - Jay Fuller
>> To: AnimalFarm Microwave Users Group
>> Sent: Monday, October 4, 2021 4:44 PM
>> Subject: Re: [AFMUG] facebook outage
>>  
>>  
>> yup.  resolving here.  but nothing coming up yet.
>>  
>> - Original Message -
>> From: Mike Hammett
>> To: AnimalFarm Microwave Users Group
>> Sent: Monday, October 4, 2021 4:36 PM
>> Subject: Re: [AFMUG] facebook outage
>>  
>> They're back...  ish.
>> 
>> 
>> 
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> <~WRD0002.jpg>
>> <~WRD0002.jpg>
>> <~WRD0002.jpg>
>> <~WRD0002.jpg>
>> 
>> Midwest Internet Exchange
>> <~WRD0002.jpg>
>> <~WRD0002.jpg>
>> <~WRD0002.jpg>
>> 
>> The Brothers WISP
>> <~WRD0002.jpg>
>> <~WRD0002.jpg>
>> 
>> 
>> 
>> From: "CBB - Jay Fuller" 
>> To: af@af.afmug.com
>> Sent: Monday, October 4, 2021 1:13:01 PM
>> Subject: [AFMUG] facebook outage
>> 
>>  
>> anyone else feeling alone and isolated without facebook?  I need my safe 
>> space (no, not myspace) back.
>>  
>> babaahahah.
>>  
>>  
>>  
>> 
>> -- 
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>  
>> -- 
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>> -- 
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>> -- 
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>> 
>> 
> -- 
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] facebook outage

2021-10-05 Thread Bill Prince 

  
  
Heard the Zuch got around $6 or $7 billion trimmed from his net
  worth yesterday. Now he's only worth around $121 billion.


bp

On 10/5/2021 6:52 AM,
  dmmoff...@gmail.com wrote:


  
  
  
  
  
I am very curious what it was that took 6
  hours to figure out or undo.
 
…..but I wouldn’t judge too harshly.  When
  any contraption gets big and complex it’s easy to overlook one
  seemingly small thing.
 
Remember this XKCD?
 

 
 
 

  From: AF
 On Behalf Of Steve
Jones
Sent: Monday, October 04, 2021 6:51 PM
To: AnimalFarm Microwave Users Group

Subject: Re: [AFMUG] facebook outage

 

  so it turns out FB murdered the
announcements for the DNS server prefixes, including the
registrar.
  
https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
  
  
 
  
  
This really seems like an
  elementary design flaw. Even a moron like me knows not to
  poke the DNS subnets all at once, and we probably get way
  less than 1k daily lookups across all the domains we host
  DNS for. Ill bet nobody involved in this makes less than
  80k a year. facebook lost 4% stock value today attributed
  to this on top of the almost percent attributed to the
  "whistleblower". I get nervous when I lose the company a
  number 10 wrench.
  

 

  
On Mon, Oct 4, 2021 at 4:51 PM CBB -
  Jay Fuller 
  wrote:
  
  

  
 
  
  
it
is alive here again
  
  
 
  
  

  -
  Original Message - 


  From:
  CBB - Jay Fuller 


  To: AnimalFarm
Microwave Users Group 


  Sent:
  Monday, October 4, 2021 4:44 PM


  Subject: Re:
  [AFMUG] facebook outage


   


   


  yup. 
  resolving here.  but nothing coming up yet.


   


  
-
Original Message - 
  
  
From:
Mike Hammett 
  
  
To: AnimalFarm
  Microwave Users Group 
  
  
Sent:
Monday, October 4, 2021 4:36 PM
  
  
Subject: Re:
[AFMUG] facebook outage
  
  
 
  
  
They're
back...  ish.

  
  
  -
  Mike Hammett
Intelligent
  Computing Solutions

Midwest
  Internet Exchange

The
  Brothers WISP

  
  





  From:
  "CBB
  - Jay Fuller" 
  To: af@af.afmug.com
  Sent: Monday, October 4, 2021 1:13:01
  PM
  Subject: [AFMUG] facebook outage
  
 
  
  
anyone
else feeling alone and isolated without
facebook?  I need my safe space (no, not

Re: [AFMUG] facebook outage

2021-10-05 Thread dmmoffett 
I am very curious what it was that took 6 hours to figure out or undo.

 

…..but I wouldn’t judge too harshly.  When any contraption gets big and complex 
it’s easy to overlook one seemingly small thing.

 

Remember this XKCD?

 



 

 

 

From: AF  On Behalf Of Steve Jones
Sent: Monday, October 04, 2021 6:51 PM
To: AnimalFarm Microwave Users Group 
Subject: Re: [AFMUG] facebook outage

 

so it turns out FB murdered the announcements for the DNS server prefixes, 
including the registrar.

https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ

 

This really seems like an elementary design flaw. Even a moron like me knows 
not to poke the DNS subnets all at once, and we probably get way less than 1k 
daily lookups across all the domains we host DNS for. Ill bet nobody involved 
in this makes less than 80k a year. facebook lost 4% stock value today 
attributed to this on top of the almost percent attributed to the 
"whistleblower". I get nervous when I lose the company a number 10 wrench.

 

On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller mailto:par...@cyberbroadband.net> > wrote:

 

it is alive here again

 

- Original Message - 

From: CBB - Jay Fuller   

To: AnimalFarm Microwave Users Group   

Sent: Monday, October 4, 2021 4:44 PM

Subject: Re: [AFMUG] facebook outage

 

 

yup.  resolving here.  but nothing coming up yet.

 

- Original Message - 

From: Mike Hammett   

To: AnimalFarm Microwave Users Group   

Sent: Monday, October 4, 2021 4:36 PM

Subject: Re: [AFMUG] facebook outage

 

They're back...  ish.



-
Mike Hammett
  Intelligent Computing Solutions
   
  
  
 
  Midwest Internet Exchange
   
  
 
  The Brothers WISP
   
 





  _  


From: "CBB - Jay Fuller" mailto:par...@cyberbroadband.net> >
To: af@af.afmug.com  
Sent: Monday, October 4, 2021 1:13:01 PM
Subject: [AFMUG] facebook outage

 

anyone else feeling alone and isolated without facebook?  I need my safe space 
(no, not myspace) back.

 

babaahahah.

 

 

 


-- 
AF mailing list
AF@af.afmug.com  
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

 

  _  

-- 
AF mailing list
AF@af.afmug.com  
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

  _  

-- 
AF mailing list
AF@af.afmug.com  
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

-- 
AF mailing list
AF@af.afmug.com  
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com