Send Outages-discussion mailing list submissions to
outages-discussion@outages.org
To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/outages-discussion
or, via email, send a message with subject or body 'help' to
outages-discussion-requ...@outages.org
You can reach the person managing the list at
outages-discussion-ow...@outages.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages-discussion digest..."
Today's Topics:
1. Re: [EXTERNAL] Re: [outages] Facebook Down (Jeffrey Ollie)
2. Re: [outages] Facebook.com no records? (Leato, Gary)
3. Re: [outages] Facebook Down (DaZZa)
4. What's Happening at Facebook, etc. (Matt Barton)
----------------------------------------------------------------------
Message: 1
Date: Mon, 4 Oct 2021 13:23:50 -0500
From: Jeffrey Ollie <j...@ocjtech.us>
To: "Chapman, Brad (NBCUniversal)" <brad.chap...@nbcuni.com>
Cc: outages-discussion <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [EXTERNAL] Re: [outages] Facebook
Down
Message-ID:
<CAD-DQxTzZPwOKBzQn=8zRqbctqFWEZh=f7ptcrwvkibquuj...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/
I think keeping the comments secret is no longer possible.
On Mon, Oct 4, 2021 at 1:17 PM Chapman, Brad (NBCUniversal) <
brad.chap...@nbcuni.com> wrote:
> Here's a partial comment chain from the reddit /r/sysadmin forum, started
> by a user who claims to be a part of the "recovery and investigation team"
> at Facebook.
>
> Also... in the last few seconds this person just deleted their comments so
> he probably doesn't want this shared publicly. Keep it in the mailing
> list, please.
>
> =============
>
> (fb person)
>
> Update 1640 UTC:
>
> As many of you know, DNS for FB services has been affected and this is
> likely a symptom of the actual issue, and that's that BGP peering with
> Facebook peering routers has gone down, very likely due to a configuration
> change that went into effect shortly before the outages happened (started
> roughly 1540 UTC).
>
> There are people now trying to gain access to the peering routers to
> implement fixes, but the people with physical access is separate from the
> people with knowledge of how to actually authenticate to the systems and
> people who know what to actually do, so there is now a logistical challenge
> with getting all that knowledge unified.
>
> Part of this is also due to lower staffing in data centers due to pandemic
> measures.
>
> > (reply) "That implies access was lost that wasn't planned... was this
> malicious?"
>
> >> (fb person) No discussion that I'm aware of yet that is considering a
> threat/attack vector. I believe the original change was 'automatic' (as in
> configuration done via a web interface). However, now that connection to
> the outside world is down, remote access to those tools don't exist
> anymore, so the emergency procedure is to gain physical access to the
> peering routers and do all the configuration locally.
>
> >>> (reply) "Even in the biggest of organizations, they still have to wait
> for somebody to race down to the datacenter and plug his laptop into a
> router."
>
> =============
>
>
> ?On 10/4/21, 10:36 AM, "Outages on behalf of Jay Weekley via Outages" <
> outages-boun...@outages.org on behalf of outa...@outages.org> wrote:
>
> Same in North Alabama. The one time in my history of Facebook that I
> need to get in touch with someone urgently.
>
> Matt Hoppes via Outages wrote:
> > Is anyone else seeing Facebook down? Seems to be offline here in
> > Pennsylvania.. started with error messages now DNS errors.
> >
> > Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
> > _______________________________________________
> > Outages mailing list
> > outa...@outages.org
> >
> https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2qvc9T3xA$
> >
>
> --
> *Jay Weekley*
> *Cyber Broadband
> *
>
> --
> This email has been checked for viruses by AVG.
>
> https://urldefense.com/v3/__https://www.avg.com__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2rqgIkaYA$
>
> _______________________________________________
> Outages mailing list
> outa...@outages.org
>
> https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2qvc9T3xA$
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
--
Jeff Ollie
The majestik m??se is one of the m?ni interesting furry animals in Sweden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20211004/3e13babc/attachment-0001.htm>
------------------------------
Message: 2
Date: Mon, 4 Oct 2021 18:45:17 +0000
From: "Leato, Gary" <gle...@advance-trading.com>
To: Ross Halliday <ross.halli...@wtccommunications.ca>,
"outages-discussion@outages.org" <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Facebook.com no records?
Message-ID:
<ch2pr05mb701633585b66be129c8067d095...@ch2pr05mb7016.namprd05.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Thanks for the clarification Ross. Seems I and lots of others jumped the gun to
a total bgp peer offline, when in actuality it does seem that there was alot of
route announcements and withdrawals directly preceding the outage start.
Interesting to know about the nsa and nsb hosts though..... will definitely be
interesting to see how this all shakes out and how the pieces fit together.
-----Original Message-----
From: Ross Halliday <ross.halli...@wtccommunications.ca>
Sent: Monday, October 4, 2021 1:22 PM
To: Leato, Gary <gle...@advance-trading.com>; outages-discussion@outages.org
Subject: Re: [Outages-discussion] [outages] Facebook.com no records?
> The dns issues at hand are a symptom of the larger issues. Facebook's bgp
> peerings have disappeared globally.
Perhaps in some parts of the world, however AS32934 is still up over TorIX
(Toronto, CA) and even announcing 129.134.0.0/17, where a.ns.facebook.com and
b.ns.facbook.com reside (129.134.30.12 and 129.134.31.12, respectively). Traces
to UDP/53 die within FB's network, so the problem is further inward
- Ross
-----Original Message-----
From: Outages-discussion <outages-discussion-boun...@outages.org> On Behalf Of
Leato, Gary
Sent: October 4, 2021 1:07 PM
To: outages-discussion@outages.org
Subject: [EXTERNAL] Re: [Outages-discussion] [outages] Facebook.com no records?
Moving this to discussion group.
The dns issues at hand are a symptom of the larger issues. Facebook's bgp
peerings have disappeared globally.
The current working theory is that a well meaning but misguided automated
process removed all of the nameserv records from dns once the servers were no
longer reachable due to the bgp issues.
This is not going to be a quick recovery, especially by facebook standards.
The risk of trading futures and options can be substantial. All information,
publications, and material used and distributed by Advance Trading Inc. shall
be construed as a solicitation. ATI does not maintain an independent research
department as defined in CFTC Regulation 1.71. Information obtained from
third-party sources is believed to be reliable, but its accuracy is not
guaranteed by Advance Trading Inc. Past performance is not necessarily
indicative of future results.
------------------------------
Message: 3
Date: Tue, 5 Oct 2021 07:41:11 +1100
From: DaZZa <dazzagi...@gmail.com>
To: Bryan Holloway <br...@shout.net>
Cc: Matt Hoppes <mattli...@rivervalleyinternet.net>,
"outages-discussion@outages.org" <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Facebook Down
Message-ID:
<CAMpV8MphpLa9MJ6MVhNuE=7sdf41SOt=_iqt4buyzpdcvvg...@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Apparently, a BGP hijack of the DNS segments by a mob called eNet (AS10297)
https://twitter.com/DougMadory/status/988966914675404800
Advertising more specific routes for the segments - viola, no more Facebook.
Depending on your point of view, either a work of genius to rid the
world of Facebook, or an innocent mistake that's gonna get someone
fired/sued.
DaZZa
On Tue, 5 Oct 2021 at 07:07, Bryan Holloway <br...@shout.net> wrote:
>
> Oops ...
>
> https://dnsviz.net/d/facebook.com/dnssec/
>
> D?j? vu?
>
>
> On 10/4/21 5:46 PM, Matt Hoppes via Outages wrote:
> > Is anyone else seeing Facebook down? Seems to be offline here in
> > Pennsylvania.. started with error messages now DNS errors.
> >
> > Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
> > _______________________________________________
> > Outages mailing list
> > outa...@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
--
veg?e?tar?i?an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
------------------------------
Message: 4
Date: Mon, 4 Oct 2021 17:15:50 -0400
From: Matt Barton <m...@netmonkey.net>
To: outages-discussion@outages.org
Subject: [Outages-discussion] What's Happening at Facebook, etc.
Message-ID: <d942b66f-3427-45b4-a3e7-5115ecfd5...@netmonkey.net>
Content-Type: text/plain; charset=us-ascii
Hello
It seems it's BGP.
https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/
I can't wait for the postmortem.
--
Matt Barton
m...@netmonkey.net
------------------------------
Subject: Digest Footer
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
End of Outages-discussion Digest, Vol 140, Issue 4
**************************************************
