Re: massive facebook outage presently

[Miles Fidelman]

Jay Hennigan wrote:

On 10/4/21 09:27, Tom Beecher wrote:
Curious if there is a malicious angle after the 60 Minutes story last 
night.


Hanlon's razor.

"Never attribute to malice that which is adequately explained by 
stupidity"


Corollary - "But don't rule out malice."


"Once is happenstance. Twice is coincidence. The third*time it's*enemy 
action'.”


--
In theory, there is no difference between theory and practice.
In practice, there is.   Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

Re: massive facebook outage presently

[Aaron C. de Bruyn via NANOG]

It looks like it might take a while according to a news reporter's tweet:

"Was just on phone with someone who works for FB who described employees
unable to enter buildings this morning to begin to evaluate extent of
outage because their badges weren’t working to access doors."

https://twitter.com/sheeraf/status/1445099150316503057?s=20

-A

On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:

> I am starting to see reports that in ISPs with very large numbers of
> residential users, customers are starting to press the factory-reset
> buttons on their home routers/modems/whatever, in an attempt to make
> Facebook work. This is resulting in much heavier than normal first tier
> support volumes. The longer it stays down the worse this is going to get.
>
>
>
> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>
>> On 10/4/21 12:11, b...@theworld.com wrote:
>> >
>> > Although I believe it's generally true that if a company appears
>> > prominently in the news it's liable to be attacked I assume because
>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>> > motivation w/o some evidence like even a credible twitter post (maybe
>> > they posted that on FB? :-)
>>
>> I personally believe that the outage was caused by human error and not
>> something malicious. Time will tell.
>>
>> However, if you missed the 60 Minutes piece, it was a former employee
>> who spoke out with some rather powerful observations. I don't think that
>> this type of worldwide outage was caused by an outside bad actor. It is
>> certainly within the realm of possibility that it was an inside job.
>>
>> In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>> --
>> Jay Hennigan - j...@west.net
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>
>

Re: massive facebook outage presently

[Mel Beckman]

I’m not the only one who finds this timing suspicious, Starting with the 
publishers of 60 Minutes themselves :-)

CBS:

The outage comes the morning after "60 Minutes" aired an interview with a 
whistleblower who said Facebook is aware of how it amplifies hate, 
misinformation and unrest but claimed the company hides what it knows.

https://www.cbsnews.com/news/facebook-instagram-whatsapp-down-2021-10-04/

https://abcnews.go.com/Technology/facebook-instagram-users-us/story?id=80397437

https://www.cnbc.com/2021/10/04/facebook-shares-drop-5percent-after-site-outage-and-whistleblower-interview.html

https://www.insidenova.com/headlines/facebook-instagram-down-after-60-minutes-whistleblower-report/article_29977530-2531-11ec-b5ae-73311c46edfb.html

https://adage.com/article/digital-marketing-ad-tech-news/what-facebook-telling-advertisers-about-60-minutes-whistleblower/2370346




 -mel beckman

On Oct 4, 2021, at 1:36 PM, Blake Dunlap  wrote:


If there isn't an undernetwork capable of being backdoored with the proper keys 
(I'd be shocked if there isn't - the big players have very good infra and DR 
people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't even 
get to the locks without getting on a plane.

Regardless of how people may feel about the company, I just feel bad for their 
sres right now and am drinking one in their honor. I just want to know what an 
October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl 
mailto:baldur.nordd...@gmail.com>> wrote:
Not in such a primitive fashion no. But they could definitely have a secondary 
network that will continue to work even if something goes wrong with the 
primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli 
mailto:pjcape...@pm.me>> wrote:
Seems unlikely that FB internal controls would allow such a backdoor ...

"Never to get lost, is not living" - Rebecca Solnit

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl 
mailto:baldur.nordd...@gmail.com>> wrote:


On Mon, 4 Oct 2021 at 21:58, Michael Thomas 
mailto:m...@mtcc.com>> wrote:


On 10/4/21 11:48 AM, Luke Guillory wrote:


I believe the original change was 'automatic' (as in configuration done via a 
web interface). However, now that connection to the outside world is down, 
remote access to those tools don't exist anymore, so the emergency procedure is 
to gain physical access to the peering routers and do all the configuration 
locally.

Assuming that this is what actually happened, what should fb have done 
different (beyond the obvious of not screwing up the immediate issue)? This 
seems like it's a single point of failure. Should all of the BGP speakers have 
been dual homed or something like that? Or should they not have been mixing ops 
and production networks? Sorry if this sounds dumb.

Facebook is a huge network. It is doubtful that what is going on is this 
simple. So I will make no guesses to what Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using 
someone else's network. Nowadays this could be a simple 4/5G router with a VPN, 
to a terminal server that allows the operator to configure the equipment 
through the monitor port even when the config is completely destroyed.

Regards,

Baldur

RE: massive facebook outage presently

[Jean St-Laurent via NANOG]

The glue records for the NS are set at 48 hours.

 

dig @c.gtld-servers.net. facebook.com. NS

;;

facebook.com.   172800  IN  NS  a.ns.facebook.com.

facebook.com.   172800  IN  NS  b.ns.facebook.com.

facebook.com.   172800  IN  NS  c.ns.facebook.com.

facebook.com.   172800  IN  NS  d.ns.facebook.com.

 

What happens if the NS aren’t back within 48 hours? 

 

Jean

 

From: NANOG  On Behalf Of Eric Kuhnke
Sent: October 4, 2021 4:33 PM
To: Jay Hennigan ; nanog@nanog.org list 
Subject: Re: massive facebook outage presently

 

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset buttons on 
their home routers/modems/whatever, in an attempt to make Facebook work. This 
is resulting in much heavier than normal first tier support volumes. The longer 
it stays down the worse this is going to get.

 

 

 

On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan mailto:j...@west.net> > wrote:

On 10/4/21 12:11, b...@theworld.com <mailto:b...@theworld.com>  wrote:
> 
> Although I believe it's generally true that if a company appears
> prominently in the news it's liable to be attacked I assume because
> the miscreants sit around thinking "hmm, who shall we attack today oh
> look at that shiny headline!" I'd hate to ascribe any altruistic
> motivation w/o some evidence like even a credible twitter post (maybe
> they posted that on FB? :-)

I personally believe that the outage was caused by human error and not 
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee 
who spoke out with some rather powerful observations. I don't think that 
this type of worldwide outage was caused by an outside bad actor. It is 
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net <mailto:j...@west.net> 
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 18:22, Eric Kuhnke wrote:

Considering the massive impact of this it would be interesting to see 
some traffic graphs from ISPs that have PNIs with Facebook, or high 
volume peering sessions across an IX, showing traffic to FB falling 
off a cliff.


Our PNI's and FNA's in Africa have dipped.

But PNI's in Europe have nearly doubled, even though we aren't 
delivering any Facebook services of note to our eyeballs.


Strangest thing...

Mark.

Re: massive facebook outage presently

[Matt Hoppes]

Yes,
We've seen that.

On 10/4/21 4:33 PM, Eric Kuhnke wrote:
I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first tier 
support volumes. The longer it stays down the worse this is going to get.




On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan > wrote:


On 10/4/21 12:11, b...@theworld.com  wrote:
 >
 > Although I believe it's generally true that if a company appears
 > prominently in the news it's liable to be attacked I assume because
 > the miscreants sit around thinking "hmm, who shall we attack today oh
 > look at that shiny headline!" I'd hate to ascribe any altruistic
 > motivation w/o some evidence like even a credible twitter post (maybe
 > they posted that on FB? :-)

I personally believe that the outage was caused by human error and not
something malicious. Time will tell.

However, if you missed the 60 Minutes piece, it was a former employee
who spoke out with some rather powerful observations. I don't think
that
this type of worldwide outage was caused by an outside bad actor. It is
certainly within the realm of possibility that it was an inside job.

In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

-- 
Jay Hennigan - j...@west.net 

Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

[None None]

Not all employees are having this issue
But it is curious why the records can be moved to the name servers that are
still being advertised over the Internet still .


On Mon, Oct 4, 2021 at 4:53 PM Aaron C. de Bruyn via NANOG 
wrote:

> It looks like it might take a while according to a news reporter's tweet:
>
> "Was just on phone with someone who works for FB who described employees
> unable to enter buildings this morning to begin to evaluate extent of
> outage because their badges weren’t working to access doors."
>
> https://twitter.com/sheeraf/status/1445099150316503057?s=20
>
>
> -A
>
> On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:
>
>> I am starting to see reports that in ISPs with very large numbers of
>> residential users, customers are starting to press the factory-reset
>> buttons on their home routers/modems/whatever, in an attempt to make
>> Facebook work. This is resulting in much heavier than normal first tier
>> support volumes. The longer it stays down the worse this is going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>>
>>> On 10/4/21 12:11, b...@theworld.com wrote:
>>> >
>>> > Although I believe it's generally true that if a company appears
>>> > prominently in the news it's liable to be attacked I assume because
>>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>>> > motivation w/o some evidence like even a credible twitter post (maybe
>>> > they posted that on FB? :-)
>>>
>>> I personally believe that the outage was caused by human error and not
>>> something malicious. Time will tell.
>>>
>>> However, if you missed the 60 Minutes piece, it was a former employee
>>> who spoke out with some rather powerful observations. I don't think that
>>> this type of worldwide outage was caused by an outside bad actor. It is
>>> certainly within the realm of possibility that it was an inside job.
>>>
>>> In other news:
>>>
>>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>>
>>> --
>>> Jay Hennigan - j...@west.net
>>> Network Engineering - CCIE #7880
>>> 503 897-8550 - WB6RDV
>>>
>>

Re: massive facebook outage presently

[Jay Hennigan]

Starting to see some prefixes return on our peering session with them in 
Los Angeles but DNS still not resolving.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: massive facebook outage presently

[Bryan Fields]

On 10/4/21 4:47 PM, Jean St-Laurent via NANOG wrote:
>  
> dig @c.gtld-servers.net. facebook.com. NS
> ;;
> facebook.com.   172800  IN  NS  a.ns.facebook.com.
> facebook.com.   172800  IN  NS  b.ns.facebook.com.
> facebook.com.   172800  IN  NS  c.ns.facebook.com.
> facebook.com.   172800  IN  NS  d.ns.facebook.com.
> 
> What happens if the NS aren’t back within 48 hours?

Facebook can just update it at the registrar.

$ whois facebook.com
   Domain Name: FACEBOOK.COM
   Registry Domain ID: 2320948_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.registrarsafe.com
   Registrar URL: http://www.registrarsafe.com
   Updated Date: 2021-09-22T19:33:41Z
   Creation Date: 1997-03-29T05:00:00Z
   Registry Expiry Date: 2030-03-30T04:00:00Z
   Registrar: RegistrarSafe, LLC

$ dig @c.gtld-servers.net. registrarsafe.com. NS
;;
registrarsafe.com.  172800  IN  NS  a.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  b.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  c.ns.facebook.com.
registrarsafe.com.  172800  IN  NS  d.ns.facebook.com.

crap

Vertical integration is a hell of a thing.
-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

Re: massive facebook outage presently

[bzs]

17:35EDT: I'm suddenly getting:

 Sorry, something went wrong.
We're working on it and we'll get it fixed as soon as we can.

   Go Back

Facebook © 2020 · Help Center

and:

% host facebook.com

facebook.com has address 157.240.241.35
facebook.com has IPv6 address 2a03:2880:f112:182:face:b00c:0:25de
facebook.com mail is handled by 10 smtpin.vvv.facebook.com.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

RE: massive facebook outage presently

[Tony Wicks]

Back and working by the looks.

Re: massive facebook outage presently

[bzs]

One might think in over six hours they could point facebook.com's DNS
somewhere else and put up a page with some info about the outage
there, that this would be a practiced firedrill.

Yeah yeah cache blah blah but it'd get around and at least would be
coming from them. I'd imagine some mutual pact with google's or
amazon's or microsoft's cloud server(s), for example, could handle it
for a few hours, and vice-versa.

I only mean a single, simple information page like the "sorry we're
working on it" I saw just before they came back.

Then again what else would one assume?

17:44 They seem to be back, more or less, slowly loading the usual
crazy sauce.

So out about 6 hours and some.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: massive facebook outage presently

[Niels Bakker]

* b...@theworld.com (b...@theworld.com) [Tue 05 Oct 2021, 00:01 CEST]:
I only mean a single, simple information page like the "sorry we're 
working on it" I saw just before they came back.


Which would subsequently receive the world's FB session cookies.


-- Niels.

Re: massive facebook outage presently

[Doug McIntyre]

On Mon, Oct 04, 2021 at 05:50:07PM -0400, b...@theworld.com wrote:
> One might think in over six hours they could point facebook.com's DNS
> somewhere else and put up a page with some info about the outage
> there, that this would be a practiced firedrill.

Perhaps, if they didn't decide to be their own registrar as well
and run it all on the same network as it seems.

Maybe company divisions running different parts of infrastructure
should be self-hosted 100% on their own, different AS, different networking 
points, etc.
Or don't try to be everything top down all in the same company.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 22:23, Baldur Norddahl wrote:

Not in such a primitive fashion no. But they could definitely have a 
secondary network that will continue to work even if something goes 
wrong with the primary.


On IPv6, no less :-).

On a serious note, I can't even imagine what it takes to run a network 
of that scale, and plan for situations like this, if, indeed, inline 
access was lost.


Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 22:27, Łukasz Bromirski wrote:



I bet FB tested the change on smaller scale and everything was fine, 
and only then started to roll this over wider network and at that 
point „something” broke. Or some bug needed a moment to start 
cascading issues around the infra.


This is the annoyance that is our world.

In the lab, it's always good.

Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 22:33, Eric Kuhnke wrote:

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first 
tier support volumes. The longer it stays down the worse this is going 
to get.


A relative could not understand how she received an e-mail from me 
yesterday about a family matter (via GMail) when the Internet was down :-).


Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 20:48, Luke Guillory wrote:

I believe the original change was 'automatic' (as in configuration 
done via a web interface). However, now that connection to the outside 
world is down, remote access to those tools don't exist anymore, so 
the emergency procedure is to gain physical access to the peering 
routers and do all the configuration locally.




Q: What is automation?
A: Breaking the network at scale.

I suppose we shall know more in the coming days, but to me, it smells 
like a wide-scale automatic update roll-out that didn't go to plan. 
Isn't the first time a major content provider has suffered this; likely 
won't be the last.


In the end, the best thing for all of us is that it is a teaching 
moment, and we move the needle forward.


Mark.

Re: massive facebook outage presently

[Karl Auer]

On Tue, 2021-10-05 at 06:31 +0200, Mark Tinka wrote:
> Q: What is automation?
> A: Breaking the network at scale.

P J Plauger (I think) once defined a computer as a mechanism allowing
the deletion of vast quantities of irreplaceable data using simple
mnemonic commands. He defined a network as a mechanism allowing the
deletion of vast quantities of *other people's* irreplaceable data
using simple mnemonic commands...

Regards, K.

-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

Re: massive facebook outage presently

[PJ Capelli via NANOG]

Seems unlikely that FB internal controls would allow such a backdoor ...

"Never to get lost, is not living" - Rebecca Solnit

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl 
 wrote:

> On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:
> 

> > On 10/4/21 11:48 AM, Luke Guillory wrote:
> > 

> > > I believe the original change was 'automatic' (as in configuration done 
> > > via a web interface). However, now that connection to the outside world 
> > > is down, remote access to those tools don't exist anymore, so the 
> > > emergency procedure is to gain physical access to the peering routers and 
> > > do all the configuration locally.
> > 

> > Assuming that this is what actually happened, what should fb have done 
> > different (beyond the obvious of not screwing up the immediate issue)? This 
> > seems like it's a single point of failure. Should all of the BGP speakers 
> > have been dual homed or something like that? Or should they not have been 
> > mixing ops and production networks? Sorry if this sounds dumb.
> 

> Facebook is a huge network. It is doubtful that what is going on is this 
> simple. So I will make no guesses to what Facebook is or should be doing.
> 

> However the traditional way for us small timers is to have a backdoor using 
> someone else's network. Nowadays this could be a simple 4/5G router with a 
> VPN, to a terminal server that allows the operator to configure the equipment 
> through the monitor port even when the config is completely destroyed.
> 

> Regards,
> 

> Baldur

signature.asc
Description: OpenPGP digital signature

Re: massive facebook outage presently

[Jorge Amodio]

How come such a large operation does not have an out of bound access in
case of emergencies ???

Somebody's getting fired !

-J


On Mon, Oct 4, 2021 at 3:51 PM Aaron C. de Bruyn via NANOG 
wrote:

> It looks like it might take a while according to a news reporter's tweet:
>
> "Was just on phone with someone who works for FB who described employees
> unable to enter buildings this morning to begin to evaluate extent of
> outage because their badges weren’t working to access doors."
>
> https://twitter.com/sheeraf/status/1445099150316503057?s=20
>
> -A
>
> On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:
>
>> I am starting to see reports that in ISPs with very large numbers of
>> residential users, customers are starting to press the factory-reset
>> buttons on their home routers/modems/whatever, in an attempt to make
>> Facebook work. This is resulting in much heavier than normal first tier
>> support volumes. The longer it stays down the worse this is going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>>
>>> On 10/4/21 12:11, b...@theworld.com wrote:
>>> >
>>> > Although I believe it's generally true that if a company appears
>>> > prominently in the news it's liable to be attacked I assume because
>>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>>> > motivation w/o some evidence like even a credible twitter post (maybe
>>> > they posted that on FB? :-)
>>>
>>> I personally believe that the outage was caused by human error and not
>>> something malicious. Time will tell.
>>>
>>> However, if you missed the 60 Minutes piece, it was a former employee
>>> who spoke out with some rather powerful observations. I don't think that
>>> this type of worldwide outage was caused by an outside bad actor. It is
>>> certainly within the realm of possibility that it was an inside job.
>>>
>>> In other news:
>>>
>>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>>
>>> --
>>> Jay Hennigan - j...@west.net
>>> Network Engineering - CCIE #7880
>>> 503 897-8550 - WB6RDV
>>>
>>

Re: massive facebook outage presently

[David Andrzejewski]

I find it hilarious and ironic that their CTO had to use a competitor’s 
platform to confirm their outage. 


- dave

> On Oct 4, 2021, at 16:45, Hank Nussbacher  wrote:
> 
> On 04/10/2021 22:05, Jason Kuehl wrote:
> 
> BGP related:
> https://twitter.com/SGgrc/status/1445116435731296256
> as also related by FB CTO:
> https://twitter.com/atoonk/status/1445121351707070468
> 
> -Hank
> 
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20 
>> 
>> On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks > > wrote:
>>Didn't write that part of the automation script and that coder left...
>> > I got a mail that Facebook was leaving NLIX. Maybe someone
>>botched the
>> > script so they took down all BGP sessions instead of just NLIX
>>and now
>> > they can't access the equipment to put it back... :-)
>> -- 
>> Sincerely,
>> Jason W Kuehl
>> Cell 920-419-8983
>> jason.w.ku...@gmail.com 
> 

Re: massive facebook outage presently

[Glenn Kelley]

This is why you should have Routers that are Firmware Defaulted to your 
own network.  ALWAYS


Be it Calix or even a Mikrotik which you have setup with Netboot - 
having these default to your own setup is REALLY a game changer.


Without it - you are rolling trucks or at minimum taking heavy call 
volumes.



Glenn Kelley
Chief cook and Bottle Washing Watcher @ Connectivity.Engineer

On 10/4/2021 3:56 PM, Matt Hoppes wrote:

Yes,
We've seen that.

On 10/4/21 4:33 PM, Eric Kuhnke wrote:
I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first 
tier support volumes. The longer it stays down the worse this is 
going to get.




On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan > wrote:


    On 10/4/21 12:11, b...@theworld.com  wrote:
 >
 > Although I believe it's generally true that if a company appears
 > prominently in the news it's liable to be attacked I assume 
because
 > the miscreants sit around thinking "hmm, who shall we attack 
today oh

 > look at that shiny headline!" I'd hate to ascribe any altruistic
 > motivation w/o some evidence like even a credible twitter post 
(maybe

 > they posted that on FB? :-)

    I personally believe that the outage was caused by human error 
and not

    something malicious. Time will tell.

    However, if you missed the 60 Minutes piece, it was a former 
employee

    who spoke out with some rather powerful observations. I don't think
    that
    this type of worldwide outage was caused by an outside bad actor. 
It is

    certainly within the realm of possibility that it was an inside job.

    In other news:

https://twitter.com/disclosetv/status/1445100931947892736?s=20

    --     Jay Hennigan - j...@west.net 
    Network Engineering - CCIE #7880
    503 897-8550 - WB6RDV

RE: massive facebook outage presently

[Jean St-Laurent via NANOG]

I don't understand how this would have helped yesterday.

>From what is public so far, they really paint themselves in a corner with no 
>way out. A classic, but at epic scale.

They will learn and improve for sure, but I don't understand how "firmware 
default to your own network" would have help here.

Can you elaborate a bit please?

Jean

-Original Message-
From: NANOG  On Behalf Of Glenn Kelley
Sent: October 4, 2021 8:18 PM
To: nanog@nanog.org
Subject: Re: massive facebook outage presently

This is why you should have Routers that are Firmware Defaulted to your own 
network.  ALWAYS

Be it Calix or even a Mikrotik which you have setup with Netboot - having these 
default to your own setup is REALLY a game changer.

Without it - you are rolling trucks or at minimum taking heavy call volumes.


Glenn Kelley
Chief cook and Bottle Washing Watcher @ Connectivity.Engineer

On 10/4/2021 3:56 PM, Matt Hoppes wrote:
> Yes,
> We've seen that.
>
> On 10/4/21 4:33 PM, Eric Kuhnke wrote:
>> I am starting to see reports that in ISPs with very large numbers of 
>> residential users, customers are starting to press the factory-reset 
>> buttons on their home routers/modems/whatever, in an attempt to make 
>> Facebook work. This is resulting in much heavier than normal first 
>> tier support volumes. The longer it stays down the worse this is 
>> going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan > <mailto:j...@west.net>> wrote:
>>
>> On 10/4/21 12:11, b...@theworld.com <mailto:b...@theworld.com> wrote:
>>  >
>>  > Although I believe it's generally true that if a company appears
>>  > prominently in the news it's liable to be attacked I assume 
>> because
>>  > the miscreants sit around thinking "hmm, who shall we attack 
>> today oh
>>  > look at that shiny headline!" I'd hate to ascribe any altruistic
>>  > motivation w/o some evidence like even a credible twitter post 
>> (maybe
>>  > they posted that on FB? :-)
>>
>> I personally believe that the outage was caused by human error 
>> and not
>> something malicious. Time will tell.
>>
>> However, if you missed the 60 Minutes piece, it was a former 
>> employee
>> who spoke out with some rather powerful observations. I don't think
>> that
>> this type of worldwide outage was caused by an outside bad actor. 
>> It is
>> certainly within the realm of possibility that it was an inside job.
>>
>> In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>> -- Jay Hennigan - j...@west.net <mailto:j...@west.net>
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>

Another Perspective - Kentik's View on the Facebook Outage

[Mark Tinka]

https://www.kentik.com/blog/facebooks-historic-outage-explained/?mkt_tok=ODY5LVBBRC04ODcAAAF_9diyPhC6WFsJNFpS4z2ggF-DWEc6FksyD3aW8B3am5tUvtTOJDYl2MIgMdsmmqOLTL-BpUugbQHAIXCT677LE0OxHM8Dy-mqRorJQXnAjg4

Mark.