Send Outages-discussion mailing list submissions to
outages-discussion@outages.org
To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/outages-discussion
or, via email, send a message with subject or body 'help' to
outages-discussion-requ...@outages.org
You can reach the person managing the list at
outages-discussion-ow...@outages.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages-discussion digest..."
Today's Topics:
1. Re: [outages] Facebook Down (Nate)
2. Re: Slow lists again too? Sent at 11:41 CDT (Jeremy Chadwick)
3. Re: [outages] Facebook Down (Chad Austin)
4. Re: [outages] Facebook Down (James Liakos)
----------------------------------------------------------------------
Message: 1
Date: Mon, 4 Oct 2021 15:47:09 -0600
From: Nate <nat...@gmail.com>
To: Rusty Dekema <rdek...@gmail.com>
Cc: Matt Hoppes <mattli...@rivervalleyinternet.net>,
outages-discussion <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Facebook Down
Message-ID:
<CABtazQL3P2CpO2O6d9uPom6C6A5goU32iqosy5ty=tuxjdq...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Wonder how many more people are going to say it's down for them, in a
certain state or country without reading this email chain?
To everyone who DOESNT READ the chain.
YES!!! Facebook, IG,Messenger, and WhatsApp are down hard, meaning no one
can access any of those services in any state or country currently.
On Mon, Oct 4, 2021, 3:39 PM Rusty Dekema <rdek...@gmail.com> wrote:
> Facebook is having DNS issues probably stemming from a network
> engineering issue:
>
> > facebook.com.
> Server: dns.google
> Address: 2001:4860:4860::8888
> *** dns.google can't find facebook.com.: Server failed
>
> > facebook.com.
> Server: dns9.quad9.net
> Address: 9.9.9.9
> *** dns9.quad9.net can't find facebook.com.: Server failed
>
> https://twitter.com/Benjojo12/status/1445054959209287680
>
> Facebook owns Instagram and WhatsApp, so this is probably responsible
> for those services' outages as well.
>
> -Rusty
>
> On Mon, Oct 4, 2021 at 12:08 PM Matt Hoppes via Outages
> <outa...@outages.org> wrote:
> >
> > Is anyone else seeing Facebook down? Seems to be offline here in
> > Pennsylvania.. started with error messages now DNS errors.
> >
> > Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
> > _______________________________________________
> > Outages mailing list
> > outa...@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20211004/840cf311/attachment-0001.htm>
------------------------------
Message: 2
Date: Mon, 4 Oct 2021 21:59:15 +0000
From: Jeremy Chadwick <j...@koitsu.org>
To: outages-discussion@outages.org
Subject: Re: [Outages-discussion] Slow lists again too? Sent at 11:41
CDT
Message-ID:
<0101017c4d517278-6ac5f913-08ac-4de2-ab8b-4a3ccf1bef1c-000...@us-west-2.amazonses.com>
Content-Type: text/plain; charset=utf-8
I'm seeing similar slowness with deliveries. I run my own
Internet-facing inbound SMTP server, so it's definitely not your
provider.
It could also be some kind of mailman or SMTP server queueing model in
place. Consider that when a major online service goes down, outages@
gets a lot of chatter.
I'm sure Jared (server owner) will investigate if applicable. :-)
--
| Jeremy Chadwick jdc_at_koitsu.org |
| UNIX Systems Administrator PGP 0x2A389531 |
| Making life hard for others since 1977. |
On Mon, Oct 04, 2021 at 11:41:33AM -0500, Andy Ringsmuth wrote:
> I suppose it could just be me, but it appears whatever is happening worldwide
> this morning with FB, etc., is mucking up outages@ and outages-discussion@
> lists too.
>
> Personally I seem to be experiencing a half hour or more of delay on both
> lists. Could be my provider possibly as well.
>
> Here?s a couple header pieces that seem to indicate a delay. Went from 11:57
> EDT to 12:30 EDT in these two segments:
>
>
> Received: from puck.nether.net (puck.nether.net [204.42.254.5]) (using
> TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519
> server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client
> certificate requested) by pdx1-sub0-mail-mx204.dreamhost.com (Postfix) with
> ESMTPS id 4HNR6W66Tfz2Mfn for <a...@andyring.com>; Mon, 4 Oct 2021 09:30:59
> -0700 (PDT)
>
> Received: from puck.nether.net (localhost [IPv6:::1]) (using TLSv1.3 with
> cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519
> server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client
> certificate requested) by puck.nether.net (Postfix) with ESMTPSA id
> 8DDC8540225; Mon, 4 Oct 2021 11:57:52 -0400 (EDT)
>
>
> ----
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> a...@andyring.com
>
> ?Better even die free, than to live slaves.? - Frederick Douglas, 1863
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
Message: 3
Date: Mon, 4 Oct 2021 18:39:50 -0400
From: Chad Austin <m...@chadaustin.org>
To: outages-discussion@outages.org
Subject: Re: [Outages-discussion] [outages] Facebook Down
Message-ID: <f683de14-7948-7162-30f7-80d3a1ecf...@chadaustin.org>
Content-Type: text/plain; charset=utf-8; format=flowed
Someone must have gone a long time before they finally learned the
importance of the reload-in command.
-Chad
On 10/4/2021 1:28 PM, Matt Hoppes wrote:
> That's bad... if I read that properly.. they've locked themselves out
> of their routers and are unable to get in without physically touching
> each one.
>
> On 10/4/21 1:00 PM, Eric Henson wrote:
>> Someone on the Facebook Incident response team is posting updates to
>> Reddit.
>>
>> https://www.reddit.com/user/ramenporn/comments/
>>
>> ?As many of you know, DNS for FB services has been affected and this
>> is likely a symptom of the actual issue, and that's that *BGP peering
>> with Facebook peering routers has gone down*, very likely due to a
>> configuration change that went into effect shortly before the outages
>> happened (started roughly 1540 UTC).
>>
>> There are people now trying to gain access to the peering routers to
>> implement fixes, but the people with physical access is separate from
>> the people with knowledge of how to actually authenticate to the
>> systems and people who know what to actually do, so there is now a
>> logistical challenge with getting all that knowledge unified.
>>
>> Part of this is also due to lower staffing in data centers due to
>> pandemic measures.?
>>
>> --------
>>
>> *Eric Henson*
>>
>> Senior Manager, IT Operations
>>
>> PFSweb, Inc.
>>
>> *o*:? 972.685.1387
>>
>> *m:*972.948.3424
>>
>> www.pfsweb.com
>>
>> *From:* Outages <outages-boun...@outages.org> *On Behalf Of *Link IP
>> via Outages
>> *Sent:* Monday, October 4, 2021 10:51 AM
>> *To:* Matt Hoppes <mattli...@rivervalleyinternet.net>
>> *Cc:* outa...@outages.org
>> *Subject:* Re: [outages] Facebook Down
>>
>> *CAUTION:*This email originated from outside of the organization. Do
>> not click links or open attachments unless you recognize the sender
>> and know the content is safe.
>>
>> Down here in? the UK as well
>>
>> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages
>> <outa...@outages.org <mailto:outa...@outages.org>> wrote:
>>
>> ??? Is anyone else seeing Facebook down?? ?Seems to be offline here in
>> ??? Pennsylvania.. started with error messages now DNS errors.
>>
>> ??? Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
>> ??? _______________________________________________
>> ??? Outages mailing list
>> ??? outa...@outages.org <mailto:outa...@outages.org>
>> ??? https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> --
>>
>> *Lance Davis*
>>
>> *Link IP Networks Limited*
>> Delta House,
>> Wavell Road,
>> Wythenshawe,
>> M22 5QZ
>>
>> Office Tel: 0161 813 1870
>>
>> Mobile Tel:?07703 196010
>>
>> Email: la...@linkipnetworks.co.uk <mailto:la...@linkipnetworks.co.uk>
>>
>> Link IP Networks Limited is a company registered in England and Wales
>> with registration number 11881958 and registered address Delta House,
>> Wavell Road, Wythenshawe, M22 5QZ.
>>
>> This e-mail and any files transmitted with it are confidential and/or
>> legally privileged and intended solely for the use of the individual
>> or entity to whom they are addressed. If you are not the addressee
>> indicated in this e-mail (or responsible for the delivery of the e
>> mail to such person) be advised that you have received this e-mail in
>> error, and that any use, dissemination, forwarding, printing, or
>> copying of this e-mail is strictly prohibited. If you have received
>> this e-mail in error please notify the sender by reply e-mail and
>> then immediately permanently delete it and all copies from your
>> system. Whilst we maintain virus checks, we accept no liability for
>> viruses introduced from this e-mail.
>>
>> ?Please consider the *environment*?before printing this e-mail
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
Message: 4
Date: Mon, 4 Oct 2021 23:07:20 +0000
From: James Liakos <ja...@liakos.com.au>
To: Chad Austin <m...@chadaustin.org>, "outages-discussion@outages.org"
<outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Facebook Down
Message-ID:
<sybpr01mb5999cf091ca5fd5f25648430ec...@sybpr01mb5999.ausprd01.prod.outlook.com>
Content-Type: text/plain; charset="windows-1252"
Or maybe commit confirmed
________________________________
From: Outages-discussion <outages-discussion-boun...@outages.org> on behalf of
Chad Austin <m...@chadaustin.org>
Sent: Tuesday, October 5, 2021 9:39:50 AM
To: outages-discussion@outages.org <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Facebook Down
[EXTERNAL EMAIL]
Someone must have gone a long time before they finally learned the
importance of the reload-in command.
-Chad
On 10/4/2021 1:28 PM, Matt Hoppes wrote:
> That's bad... if I read that properly.. they've locked themselves out
> of their routers and are unable to get in without physically touching
> each one.
>
> On 10/4/21 1:00 PM, Eric Henson wrote:
>> Someone on the Facebook Incident response team is posting updates to
>> Reddit.
>>
>> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0
>>
>> ?As many of you know, DNS for FB services has been affected and this
>> is likely a symptom of the actual issue, and that's that *BGP peering
>> with Facebook peering routers has gone down*, very likely due to a
>> configuration change that went into effect shortly before the outages
>> happened (started roughly 1540 UTC).
>>
>> There are people now trying to gain access to the peering routers to
>> implement fixes, but the people with physical access is separate from
>> the people with knowledge of how to actually authenticate to the
>> systems and people who know what to actually do, so there is now a
>> logistical challenge with getting all that knowledge unified.
>>
>> Part of this is also due to lower staffing in data centers due to
>> pandemic measures.?
>>
>> --------
>>
>> *Eric Henson*
>>
>> Senior Manager, IT Operations
>>
>> PFSweb, Inc.
>>
>> *o*: 972.685.1387
>>
>> *m:*972.948.3424
>>
>> https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0
>>
>> *From:* Outages <outages-boun...@outages.org> *On Behalf Of *Link IP
>> via Outages
>> *Sent:* Monday, October 4, 2021 10:51 AM
>> *To:* Matt Hoppes <mattli...@rivervalleyinternet.net>
>> *Cc:* outa...@outages.org
>> *Subject:* Re: [outages] Facebook Down
>>
>> *CAUTION:*This email originated from outside of the organization. Do
>> not click links or open attachments unless you recognize the sender
>> and know the content is safe.
>>
>> Down here in the UK as well
>>
>> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages
>> <outa...@outages.org <mailto:outa...@outages.org>> wrote:
>>
>> Is anyone else seeing Facebook down? Seems to be offline here in
>> Pennsylvania.. started with error messages now DNS errors.
>>
>> Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
>> _______________________________________________
>> Outages mailing list
>> outa...@outages.org <mailto:outa...@outages.org>
>>
>> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0
>>
>>
>> --
>>
>> *Lance Davis*
>>
>> *Link IP Networks Limited*
>> Delta House,
>> Wavell Road,
>> Wythenshawe,
>> M22 5QZ
>>
>> Office Tel: 0161 813 1870
>>
>> Mobile Tel: 07703 196010
>>
>> Email: la...@linkipnetworks.co.uk <mailto:la...@linkipnetworks.co.uk>
>>
>> Link IP Networks Limited is a company registered in England and Wales
>> with registration number 11881958 and registered address Delta House,
>> Wavell Road, Wythenshawe, M22 5QZ.
>>
>> This e-mail and any files transmitted with it are confidential and/or
>> legally privileged and intended solely for the use of the individual
>> or entity to whom they are addressed. If you are not the addressee
>> indicated in this e-mail (or responsible for the delivery of the e
>> mail to such person) be advised that you have received this e-mail in
>> error, and that any use, dissemination, forwarding, printing, or
>> copying of this e-mail is strictly prohibited. If you have received
>> this e-mail in error please notify the sender by reply e-mail and
>> then immediately permanently delete it and all copies from your
>> system. Whilst we maintain virus checks, we accept no liability for
>> viruses introduced from this e-mail.
>>
>> ?Please consider the *environment* before printing this e-mail
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20211004/d30b8da4/attachment.htm>
------------------------------
Subject: Digest Footer
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
End of Outages-discussion Digest, Vol 140, Issue 5
**************************************************
