[outages] STATUS UPDATE: Hurricane Electric hold CLEARED

2024-07-04 Thread Jay R. Ashworth via Outages
, we *will* expect an After-Action Report on this. Cheers, -- jra - Original Message - > From: "Jay R. Ashworth via Outages-discussion" > > To: outages@outages.org > Cc: "Outages Discussion" > Sent: Thursday, July 4, 2024 3:46:25 PM > Subject: [Outag

Re: [outages] STATUS UPDATE: Hurricane Electric ICANN Hold

2024-07-04 Thread Jay R. Ashworth via Outages
Damned keyboard. To complete: It's pretty common to have locks in your system such that you don't gratuitously turn off the electricity to a Hospital or Transit system without looking; the same sort of thing should be in effect at Network Solutions, but apparently is not. I don't expect this wi

[outages] STATUS UPDATE: Hurricane Electric ICANN Hold

2024-07-04 Thread Jay R. Ashworth via Outages
I wrote: > I have a report that the CEO of Hurricane Electric is working on this as of > 1512EDT. > > I approve of the scale of this response. :-) Turns out they've been on this for HOURS, and the top guy is not happen that NetSol isn't taking his calls: Copied from his email to NANOG: === Netwo

Re: [outages] 911 outage - MA statewide

2024-06-18 Thread Jay R. Ashworth via Outages
As noted by others, this report either has been cleared, or was "officially bogus" to begin with. AAR to follow in -discuss when I see one. Cheers, -- jra - Original Message - > From: "Jay R. Ashworth via Outages" > To: outages@outages.org > Sent: Tues

[outages] 911 outage - MA statewide

2024-06-18 Thread Jay R. Ashworth via Outages
I have a report this 3pm Tue, hour from the authenticated FB page of the Boston PD, that 9-1-1 service is out statewide in Mass; they're handling it as a Code 2 AHOD; flag down a cruiser if you need help. """ 9-11 is currently down statewide. If you are experiencing an emergency please pull a

[outages] Big Microsoft Outage, via ZDnet Thu AM

2024-05-23 Thread Jay R. Ashworth via Outages
""" An apparent Microsoft outage has impacted a slew of services, including Bing, the company's artificial intelligence assistant Copilot, ChatGPT, and more. Several Bing users reported earlier on Thursday that the company's Bing search engine was down. Those users also reported that they were h

[outages] Multi-state 911 outages reported Weds night

2024-04-17 Thread Jay R. Ashworth via Outages
At least 10 states according to this site; one back in service as of midnight. https://twitter.com/rawsalerts/status/1780797091167526930 Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think

Re: [outages] After-Action Report: Facebook/Meta outage

2024-03-05 Thread Jay R. Ashworth via Outages
uesday, March 5, 2024 2:16:33 PM > Subject: Re: [outages] After-Action Report: Facebook/Meta outage > It clearly *is* from 2010: > https://engineering.fb.com/2010/09/23/uncategorized/more-details-on-today-s-o > > Damian > > On Tue, Mar 5, 2024 at 11:13 AM Jay R. Ashworth v

Re: [outages] After-Action Report: Facebook/Meta outage

2024-03-05 Thread Jay R. Ashworth via Outages
Well, it seems equivocal whether that's about *this* outage rather than an earlier one -- though it's clearly not from 2010. More to follow. Cheers, -- jra - Original Message - > From: "Jay R. Ashworth via Outages" > To: outages@outages.org > Sent: Tues

[outages] After-Action Report: Facebook/Meta outage

2024-03-05 Thread Jay R. Ashworth via Outages
[ I can't set the header from Zimbra, but please, Reply-To: -discuss ] Here's a link to FB's AAR about this morning's 70-minuteish outage. It's been pointed out to me that this affected other services as well, if they are an OAuth client, and you chose to use FB's OAuth service to create your acc

[outages] PINCH Re: bgp.he.net down?

2024-01-16 Thread Jay R. Ashworth via Outages
Seems clear there is a problem, not necessarily affecting everyone. Please move further conversations to -discuss. Press hard, you are making 30,000 copies. This is a recording. :-) Cheers, -- jr '' a - Original Message - > From: "Hank Nussbacher via Outages" > To: "Outages" > Sen

Re: [outages] Flexential PDX (was Re: Dreamhost MySQL outage?)

2023-11-05 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Bryan Fields via Outages" > On 11/5/23 3:24 AM, Chapman, Brad (NBCUniversal) wrote: >> Counter to best practices, Flexential did not inform Cloudflare that they had >> failed over to generator power. >> >> Off to a good start, then... > > It's proper to pre

Re: [outages] IPAWS/FEMA National Test - VoWIFI

2023-10-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > I could be unsurprised to find that the carrier VoWIFI proxies missed out on > proxying SMS Cell Broadcast, which I gather provides the underlying PHY for > CMAS. I have now an authoritative report tha

[outages] IPAWS/FEMA National Test - VoWIFI

2023-10-04 Thread Jay R. Ashworth via Outages
I have a report from a usually reliable source that he did not receive today's IPAWS/CMAS/WEA test message, and he surmises this might be because his phone handset is forced to VoWIFI. I have a number of other reports, mostly though not all, from the broadcast engineering community; nobody reporte

Re: [outages] [EXTERNAL] Comcast Metro-E Chicago and Suburbs

2023-09-07 Thread Jay R. Ashworth via Outages
Unless the app provides at least a number of connections affected, and maybe an ETR and a ticket number, it's not going to be terribly useful to the people who are the audience on the Outages list -- who are largely expected to be people running networks As the old comic says: "Press hard, you are

[outages] EXEMPLAR: Re: AT&T SFO to Twitter -- possible routing issue or BGP hijack

2023-06-25 Thread Jay R. Ashworth via Outages
*This*, folks and peoples, is a good problem report. Strive to provide this much detail, or more. The price of free help is you gotta show your work: http://www.catb.org/~esr/faqs/smart-questions.html Cheers, -- jr '' a - Original Message - > From: "Jeremy Chadwick via Outages" > To:

Re: [outages] FAA.gov nameserver outage

2023-03-31 Thread Jay R. Ashworth via Outages
As is Generalissimo Francisco Franco. Indeed, folks; please move these meta conversations to the -discuss list; they are off topic for the main notification list. Cheers, -- jr '' a - Original Message - > From: "Mike Lyon via Outages" > To: "T.Suzuki" > Cc: "Michael Loftis via Outages"

Re: [outages] Twitter API problems Fri/13

2023-01-15 Thread Jay R. Ashworth via Outages
ng from some of my customers who develop Twitter clients their API >> access was revoked, and Twitter is not responding to questions why their >> access >> was revoked. >> >> Kind regards, >> >> Tom >> >> On Mon, 16 Jan 2023, at 00:55, Jay R.

Re: [outages] Twitter API problems Fri/13

2023-01-15 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > Sent: Friday, January 13, 2023 9:49:32 PM > From @intelwire: > > """ > Just to clarify, because I'm seeing a lot of bad info out there, the Twitter > API is not down. Cr

[outages] [ADMIN] Re: [Telnyx #1243282] Twitter API problems Fri/13

2023-01-13 Thread Jay R. Ashworth via Outages
Last time I replied privately (I think). This time, I won't. YO, Telnyx! If you can't keep your ticketing system from replying to posters, please unsubscribe it from the list now. Cheers, -- jr '' a - Original Message - > From: "Telnyx Support" > To: "jra" > Sent: Friday, January 13,

[outages] Twitter API problems Fri/13

2023-01-13 Thread Jay R. Ashworth via Outages
>From @intelwire: """ Just to clarify, because I'm seeing a lot of bad info out there, the Twitter API is not down. Credentials appear to have been revoked for some apps, including the biggest alternative Twitter clients. As far as I can tell, the API is still working if your credentials are wo

[outages] DC outage: Flexiential Norcross

2022-12-24 Thread Jay R. Ashworth via Outages
User in one of my facebook IT groups reports all redundant power in the DC was down for 2 hours this morning, for WX related reasons, so if you're in that DC... Merry Christmas! :-} And if you're not... Merry Christmas! Cheers, -- jra -- Jay R. Ashworth Baylink

[outages] [Preemptive Strike] Ukraine requests Internet Death Penalty for Russia

2022-03-02 Thread Jay R. Ashworth via Outages
This made the general press pretty quickly, though there's some earlier material from Woody on NANOG, for those who also subscribe there. https://www.zdnet.com/article/ukraine-asks-for-russia-to-be-kicked-off-the-internet Replies to this on -discuss, please, *as well as any conversations that amo

[outages] SLACK Outage

2022-02-22 Thread Jay R. Ashworth via Outages
Confirmed in many places, and also here; please move further discussion and reports to the -discuss list. Cheers, - jr '' a - Original Message - > From: "Paul Ferguson via Outages" > To: "Outages ML" > Sent: Tuesday, February 22, 2022 9:49:27 AM > Subject: [outages] Slack? > Apparently

[outages] Coverage of the .to outage

2022-01-18 Thread Jay R. Ashworth via Outages
This piece: https://www.npr.org/2022/01/18/1073863310/an-undersea-cable-fault-could-cut-tonga-from-the-rest-of-the-world-for-weeks drills down to this piece with slightly more detail: https://www.reuters.com/markets/funds/undersea-cable-fault-could-cut-off-tonga-rest-world-weeks-2022-01-18/ I'm

Re: [outages] not quite an outage, more a hack, "Urgent: Threat actor in systems" emails from FBI infrastructure

2021-11-13 Thread Jay R. Ashworth via Outages
- Original Message - > From: "John Sage via Outages" > On 11/13/21 8:11 AM, Glenn McGurrin via Outages wrote: >> not quite an outage, more a hack, but thought it relevant.  As always > > Is anyone besides me now receiving three (or here four) identical posts > to the list with identical

[outages] [PINCH] Experiencing VzW LTE outage across the US

2021-11-10 Thread Jay R. Ashworth via Outages
It's clear there's another (there was also one yesterday morning, across several carriers) widespread mobile data outage this morning. As usual, please remember the "Press hard, you are making 15,000 copies" rule for the main list and limit further postings to "Here's a ticket number and copy".

[outages] YouTube to force 2FA for monetized accounts next month - Gmail

2021-10-20 Thread Jay R. Ashworth via Outages
Or something very close to that. https://9to5google.com/2021/08/24/youtube-2-step-verification-requirement Directly, that's not necessarily a big thing, but if you have a gmail account associated with the YT account -- as they forced some years back -- and you've forgotten about that, devices m

[outages] GPSd bug may cause auth failures next week (10/24/2021)

2021-10-18 Thread Jay R. Ashworth via Outages
A bug in the GPS Week Rollover code of GPSd, which apparently isn't quite as tight as ESR thinks it is, may cause GPD steered clocks and infrastructre to report incorrect date-data starting on or about 10-24-2021 (presumably) UTC, and this may bubble up into authentication and security protocols (i

[outages] SSL rollover - Let's Encrypt etc

2021-10-10 Thread Jay R. Ashworth via Outages
I meant to post this when it happened, and I think I forgot. :-} The SSL Root cert that underlies Let's Encrypt's root expired on 30-Sept, and the new root that underlies it is not in the Root Certificate Package of some still pretty widely deployed OS versions, including OS/X <10.12.1. Lots of

[outages] META: What's fit for the main list

2021-10-05 Thread Jay R. Ashworth via Outages
I occasionally catch a little crap -- we've lost some subscribers for it over 8 years or more -- about what I consider a fit topic to announce on this main list. It includes outages or major impacts to what some people call "eyeball" services: things like Twitter, Google, and... Facebook. My pri

Re: [outages] Backlog cleared

2021-10-05 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jared Mauch via Outages" > Apologies for not meeting expectations. I will refund you a full month > service > fee :-) Damn well better. All seriousness aside, I want to take a moment here to thank Jared for the thankless job of running a bunch of mailin

[outages] NOTICE: List was clogged

2021-10-05 Thread Jay R. Ashworth via Outages
You're going to see a bunch of meetoos from around noon, which apparently were clogged up in the list machinery, dripping out now; please note the send time when evaluating whether we have list members who are *really* slow or not. :-) Cheers, -- jra -- Jay R. Ashworth Baylink

[outages] FACEBOOK - recovery begins after 6 hours

2021-10-05 Thread Jay R. Ashworth via Outages
You(r users) may continue to see timeouts and errors for another hour or two, into Monday evening EDT, but at least the DNS (and hence, presumably the BGP) problems appear to be fixed or in-work. As usual for an outage of this magnitude (one that makes the general press), the list won't need indiv

Re: [outages] www.facebook.com not resolving from Spectrum/Tampa

2021-10-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > Sent: Monday, October 4, 2021 11:50:51 AM > So if FB just fell off the world for you too, check with a ping. > > Time out 1145EDT or so. > > WFM/Metoo to -discuss so we don't set everyone&#

[outages] REMINDER to read before you post

2021-10-04 Thread Jay R. Ashworth via Outages
FACEBOOK and related properties are DNS-down since 1140 or so EDT. The root cause appears to be a BGP screwup, and this, from, a mailing list I follow regularly: """ I'm hearing -- NOT CONFIRMED -- that it was not an attack but was a bad BGP push that locked engineers out of remote access to syst

Re: [outages] www.facebook.com not resolving from Spectrum/Tampa

2021-10-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > So if FB just fell off the world for you too, check with a ping. > > Time out 1145EDT or so. > > WFM/Metoo to -discuss so we don't set everyone's beepers on fire. The Verge quotes someo

Re: [outages] www.facebook.com not resolving from Spectrum/Tampa

2021-10-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > So if FB just fell off the world for you too, check with a ping. > > Time out 1145EDT or so. > > WFM/Metoo to -discuss so we don't set everyone's beepers on fire. This is a wides

Re: [outages] Facebook Down

2021-10-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Matt Hoppes via Outages" > Is anyone else seeing Facebook down? Seems to be offline here in > Pennsylvania.. started with error messages now DNS errors. > > Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon. Yup; all FB properties appear to

[outages] www.facebook.com not resolving from Spectrum/Tampa

2021-10-04 Thread Jay R. Ashworth via Outages
So if FB just fell off the world for you too, check with a ping. Time out 1145EDT or so. WFM/Metoo to -discuss so we don't set everyone's beepers on fire. Cheerfs, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things

Re: [outages] AWS problem with EBS in us-east-1

2021-09-27 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Paul M - outages via Outages" > https://status.aws.amazon.com/ is reporting EBS performance problems > in us-east-1 or North Virginia. > > Not strictly an outage but it could be causing problems which may > manifest downstream as an outage with dependent ser

[outages] SF Bay Outage - Internet Archive down

2021-09-19 Thread Jay R. Ashworth via Outages
Among, presumably, lots of other stuff; datacenter outage where they're hosted: https://twitter.com/brewster_kahle/status/1439622667120754689 https://twitter.com/tvzuke/status/1439607856811020296 PG&E is, as you might expect, taking it in the nuts over how fog and mist can knock out power on t

Re: [outages] [Outages-discussion] Frontier in Thousand Oaks, CA

2021-08-16 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay Hennigan" > Our NOC finally got through to someone, but something may have been lost > in translation with regard to "fiber towers" and their ticket number > looks to be the CLLI code of One Wilshire. Also posted to -discussion, > should probably follow u

[outages] List propagation improvement

2021-07-29 Thread Jay R. Ashworth via Outages
Thanks to Jared, who runs the machinery of the Outages lists, for hammering on that machinery this week; propagation time through the list server appears to have dropped from several hours to about 5 minutes. :-) Hopefully, this will have the effect on the lists of *reducing* overlapping reports,

[outages] Propagation Test III - In 3D!

2021-07-29 Thread Jay R. Ashworth via Outages
Test 3 -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Na

[outages] Propagation Test 2 - similarly ignore

2021-07-26 Thread Jay R. Ashworth via Outages
No, really. :-) -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask Fo

[outages] Propagation Test - Do Not Reply

2021-07-23 Thread Jay R. Ashworth via Outages
No, really. -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It

Re: [outages] DNS SERVFAIL for nist.gov

2021-06-15 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Mel Beckman via Outages" > Subject: Re: [outages] DNS SERVFAIL for nist.gov > I support many financial networks, and comply with the same FINRA rule 6820 > you > do. This rule doesn’t state that the time must be synchronized over the > Internet using public

Re: [outages] Some kind of big issue on east coast?

2021-06-15 Thread Jay R. Ashworth via Outages
Indeed. Let's hit pause on this thread, folks, unless you have some quantitative information, or a ticket from a carrier to quote? As always, ask yourself: "is what I have to say worth setting off 5,000 pagers?" You can always take it to -discuss Cheers, -- jr '' a - Original Message

Re: [outages] DNS SERVFAIL for nist.gov

2021-06-14 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Matthew Huff via Outages" > Of course. > > Like I stated in the original email, we don’t use NIST for time sync. We > actually have a GPS and a PTP feed. > > WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS > VIA > FINRA REGULAT

[outages] 9-1-1 outage, NPA843 SC

2021-05-20 Thread Jay R. Ashworth via Outages
I've had a CMAS alert relayed to me this afternoon saying that 9-1-1 service is inop in Berkeley, Charleston, Colleton and Dorchester counties, South Carolina... which seem *suspiciously* alphabetical to me -- and no, that's not intended as humor. Cheers, -- jra -- Jay R. Ashworth

Re: [outages] The Internet is down.

2021-05-18 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Damian Guppy" > Sent: Wednesday, May 19, 2021 12:10:16 AM > Given the current world events, wouldn't be surprised if a Nation State was > trying to do some censorship and fat fingered their peering links. Not impossible, and part of the reason I posted it.

Re: [outages] The Internet is down.

2021-05-18 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > Sent: Tuesday, May 18, 2021 11:25:04 PM > Ok, clearly that's not what's actually happening, but nearly every graph on > the home page of downdetector.com started spiking on the same curve a

[outages] The Internet is down.

2021-05-18 Thread Jay R. Ashworth via Outages
Ok, clearly that's not what's actually happening, but nearly every graph on the home page of downdetector.com started spiking on the same curve around 2200EDT, and they've not started coming down yet; 250k+ reports for YouTube. Someone break a core IX somewhere? Cheers, -- jra -- Jay R. Ashwort

Re: [outages] Outage: the outages wiki

2021-03-02 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Rich Lafferty via Outages" > Looks like the SSL cert on wiki.outages.org expired, uh, last August. Noticed > while pointing someone else to the list. Figured a heads-up was in order. > > Relatedly, the Google Map on www.outages.org is affected by the “deprec

Re: [outages] automated response

2021-02-19 Thread Jay R. Ashworth via Outages
[ Generally, this is frowned upon, but Lenny probably has substantially bigger problems than this to deal with this week, so we won't. :-} /admin ] - Original Message - > From: "Lenny via Outages" > To: outages@outages.org > Sent: Thursday, February 18, 2021 10:41:59 PM > Subject: [outa

[outages] Fwd: Nashville

2020-12-30 Thread Jay R. Ashworth via Outages
This is from yesterday - Forwarded Message - > From: "Eric Kuhnke" > To: "Sean Donelan" > Cc: "NANOG" > Sent: Tuesday, December 29, 2020 6:06:00 PM > Subject: Re: Nashville > From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in > one place has a long recovery time wh

[outages] Google After-Action on the email 550 that *followed* the Big Outage

2020-12-25 Thread Jay R. Ashworth via Outages
PDF, sorry: http://www.google.com/appsstatus/ir/4et50yp2ckm8otv.pdf Summary: """ Background: The Gmail SMTP inbound service uses a configuration system that allows specific service options and flags to be changed while the service is already deployed in production. The "gmail.com" domain name i

[outages] Local News coverage of Nashville

2020-12-25 Thread Jay R. Ashworth via Outages
https://www.youtube.com/watch?v=iag6cTWpgq8 As usual, conversations to -discuss Cheers, -- jr '' a -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http:/

Re: [outages] Explosion in Nashville

2020-12-25 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Kurt Duncan via Outages" > Anyone else seeing phone or internet issues due to the AT&T building being > damaged by the explosion in Nashville ? Let's please continue to confine replies here to specific information about network impacts, and keep the speculat

[outages] Ongoing Gmail Outages

2020-12-15 Thread Jay R. Ashworth via Outages
After the Monday outage, it appears some statistically significant number of gmail boxes aren't accepting incoming mail -- even if they can send -- with error messages like: """ : host gmail-smtp-in.l.google.com[64.233.185.26] said: 550-5.1.1 The email account that you tried to reach does no

Re: [outages] Twitter down from Boston - and elsewhere

2020-10-15 Thread Jay R. Ashworth via Outages
Definitely regional; working fine in Florida from the web. Let's moderate the traffic until we have an actual report from them, now, shall we, folks? Cheers, -- jr '' a - Original Message - > From: "Patrick W. Gilmore via Outages" > To: "outages" > Sent: Thursday, October 15, 2020 5:45

[outages] [META] Some outages are planned

2020-09-08 Thread Jay R. Ashworth via Outages
We're prone, in the networking and IT world, to use the term "cutover" to describe what happens when we replace an old system with a new one. But even if you know where the term comes from... I'll bet you've never actually *seen* one: https://www.youtube.com/watch?v=saRir95iIWk [ reply comments

Re: [outages] [Outages-discussion] Coresite Any2 Los Angeles is in trouble

2020-09-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "scott weeks" >> On Fri, Sep 4, 2020, 4:47 PM Jay Hennigan via Outages >> mailto:outages@outages.org>> wrote: >> >> We're seeing traffic blackholed at Any2 Exchange to multiple >> destinations. Started around 19:00 UTC (Noon PDT). >> > > On 9/4/20 10:

Re: [outages] 3556 problems

2020-09-02 Thread Jay R. Ashworth via Outages
Reminder: let's move all the after-action traffic to outages-discuss. Thanks, -- jra - Original Message - > From: "Randy Bush via Outages" > To: outages@outages.org > Sent: Sunday, August 30, 2020 2:14:21 PM > Subject: Re: [outages] 3556 problems >> initial research identified that an o

[outages] [After Action] ZOOM - Regarding August 24th Service Disruption

2020-08-24 Thread Jay R. Ashworth via Outages
Well, ok, more like an apology, but there's a teeny amount of detail in this Letter From The Boss: - Forwarded Message - > From: "Zoom Video Communications" > Sent: Monday, August 24, 2020 9:37:57 PM > Subject: Regarding August 24th Service Disruption > Dear Valued Zoom Customer, > > A

[outages] AAR: Friday Cloudflare Outage

2020-07-17 Thread Jay R. Ashworth via Outages
https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/ Thanks to DaZZa for spotting this... and to everyone else, for settling down fast enough that no one read me off this time. :-) Cheers, -- jr '' a Followups-to: -discussion -- Jay R. Ashworth Baylink

[outages] Cellular Outage

2020-06-15 Thread Jay R. Ashworth via Outages
Ok, apparently this *is* affecting data. :-) This outage is affecting all 4 carriers, and in many states, according to reports in a number of channels. I have this from a Tampa local: """ This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happene

Re: [outages] Minneapolis Metro - Verizon & Centurylink?

2020-06-10 Thread Jay R. Ashworth via Outages
Concur; let's leave the editorial comment out, particularly when it isn't supported by any facts. Cheers, -- jra - Original Message - > From: "Mike Bolitho via Outages" > To: "Joey Kelly" > Cc: outages@outages.org > Sent: Sunday, June 7, 2020 2:50:36 PM > Subject: Re: [outages] Minnea

[outages] CLEAR: Facebook down regionally (FL/Central Americas)

2020-03-20 Thread Jay R. Ashworth via Outages
Original Message ----- > From: "Jay R. Ashworth via Outages" > To: outages@outages.org > Sent: Friday, March 20, 2020 2:53:16 PM > Subject: [outages] Facebook down regionally (FL/Central Americas) > Downdetector showing hotspots in Florida POPs; it's timing out for

[outages] ADMIN: General reminder

2020-03-20 Thread Jay R. Ashworth via Outages
Power and Internet are the backbones of America -- nope, no joke. Workers in these industries should take greater than usual precautions against becoming infected with the novel coronavirus; all the people stuck at home are counting on you. Stay safe, everybody. Cheers, -- jra -- Jay R. Ashwor

[outages] Facebook down regionally (FL/Central Americas)

2020-03-20 Thread Jay R. Ashworth via Outages
Downdetector showing hotspots in Florida POPs; it's timing out for me, which never happens. Caribbean apparently homes here too. Time out 1445EDT. No further. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Thing

[outages] KNOWN OUTAGE: Facebook/spam flagging

2020-03-17 Thread Jay R. Ashworth via Outages
Apparently, as Facebook itself transitions to WfH, things aren't flying as well on autopilot as they'd hoped. Starting sometime today, or last night, the codebase has been throwing lots of false positives primarily about shared articles and posts concerning... coronavirus. Isn't it ironic... do

Re: [outages] Wide-area multi-carrier NE US cell outage

2019-09-06 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jay R. Ashworth via Outages" > > What's going on is that T-Mobile, Verizon, AT&T, Sprint, and US Cellular > *ALL* have big outages in the NE corridor; all big report spikes in the > last hour or two. > > Not sure what

[outages] Wide-area multi-carrier NE US cell outage

2019-09-06 Thread Jay R. Ashworth via Outages
A friend complained on FB about T-Mobile being out at her house, and, to console her, I went to DownDetector to see what was going on. What's going on is that T-Mobile, Verizon, AT&T, Sprint, and US Cellular *ALL* have big outages in the NE corridor; all big report spikes in the last hour or two.

Re: [outages] Spectrum OH/KY towards PA

2019-08-28 Thread Jay R. Ashworth via Outages
Let's pick this up on -discussion, please. Cheers, -- jr '' a - Original Message - > From: "Mike Bolitho via Outages" > To: "Josh Luthman" > Cc: "OUTAGES Mailing List" > Sent: Wednesday, August 28, 2019 9:13:44 PM > Subject: Re: [outages] Spectrum OH/KY towards PA > You guys really ne

Re: [outages] CenturyLink Internet Outage in Iowa, possibly Nationally?

2019-08-23 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Jeffrey Ollie via Outages" > *** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 - > Event > Field Operations on site have identified that equipment laying rock in the > area > had damaged multiple cables. Efforts to identify the extent

[outages] Facebook 7/3 outage clear{ed,ing}

2019-07-03 Thread Jay R. Ashworth via Outages
Anecdotal reports and my own experience suggest that the image/live outage from this morning is now clear, or in the process of rolling out as clear. I'll post an after-action to the -discussion list when I see one. Followups to -discussion as well. Cheers, -- jra -- Jay R. Ashworth

Re: [outages] Unsubscribe

2019-07-03 Thread Jay R. Ashworth via Outages
POLICY: Noticeable outages in eyeball networks and services with lots of end-users on eyeball networks are explicitly on-topic for the list, so as to keep informed the people who run those support organizations (and keep this traffic here, instead of NANOG). :-) Cheers, -- jra - Original

Re: [outages] [EXTERNAL] Re: !!!!DYMAXION!!!!

2019-07-02 Thread Jay R. Ashworth via Outages
It may not be, but let's us not do it anyway, mmm'kay everyone? Cheers, -- jra - Original Message - > From: "Kurt H Maier via Outages" > To: "Chapman, Brad (NBCUniversal) via Outages" > Sent: Tuesday, July 2, 2019 5:26:37 PM > Subject: Re: [outages] [EXTERNAL] Re: DYMAXION >

[outages] [ADMIN] Dymaxion Nuked

2019-07-02 Thread Jay R. Ashworth via Outages
In case the email from Jared got lost in your pile, the queue has been cleared, the sender has been blacklisted, and we apologize for the hour's excitement. If anyone still wants to unsubscribe, we'll miss you and there's a link at the bottom of this message. :-} See also my policy note a few mo

[outages] POLICY: Ticketing systems

2019-07-02 Thread Jay R. Ashworth via Outages
I'm going to write this, purposely, while I'm still cranky. If you're going to route outages- into a ticketing system, you'd better make sure that *NOTHING* can leak back out to the list, or we will terminate you with extreme prejudice. I phrased it most carefully. Cheers, -- jr 'not' a -- Ja

Re: [outages] [Admin] Testing 1, 2, 3...

2019-07-02 Thread Jay R. Ashworth via Outages
It appears this fix broke the floodgates on the Dymaxion foolishness. Our apologies (to whomever's left). We are working on the problem. Cheers, -- jra - Original Message - > From: "Jay Ashworth via Outages" > To: outages@outages.org > Sent: Tuesday, July 2, 2019 4:00:03 PM > Subject:

Re: [outages] [Outages-discussion] Fwd: (JIRA) JEMH has rejected your email, 1 problem(s) were found

2019-06-24 Thread Jay R. Ashworth via Outages
Hey; give us a little credit. :-) It appears that user's been snipped. Cheers, -- jr '' a - Original Message - > From: "Jeremy Gault, KD4NED (Senior Engineer) via Outages" > > To: outages@outages.org > Sent: Monday, June 24, 2019 6:35:45 AM > Subject: Re: [outages] [Outages-discussion

Re: [outages] AT&T Hawaii can't call Lihue rate center numbers hosted by Verizon.

2019-06-16 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Aryn Nakaoka 808.356.2901 via Outages" > How would you guys go about fixing this issue? > > @ATT phones can't call Kauai numbers hosted by Verizon. Verizon is saying the > calls are not leaving AT&T *Watching the SS7 traffic. AT&T phones on the > mainland ca

[outages] ADMIN: Fwd: (JIRA) JEMH has rejected your email, 1 problem(s) were found

2019-06-13 Thread Jay R. Ashworth via Outages
- Forwarded Message - > From: nore...@atlassian.endurance.com > To: "jra" > Sent: Friday, June 14, 2019 12:30:56 AM > Subject: (JIRA) JEMH has rejected your email, 1 problem(s) were found > Hello, > > Some of the fields provided were not valid. Please review the following 1 > hints for

[outages] Instagram is DOWN; PSN too

2019-06-13 Thread Jay R. Ashworth via Outages
Instagram -- and PlayStation Network -- are both hard down tonight, according to a story in the Independent. People who run eyeball-network support organizations may want to broadcast this on your internal whiteboards. Let's limit further posting here to people who *know what happened*, mmm'kay?

Re: [outages] Verizon outage

2018-10-29 Thread Jay R. Ashworth via Outages
Downdetector says it's peaked, and markets are coming back up: https://downdetector.com/status/verizon We'll keep our eyes peeled for an after-action, and post it to -discuss when available. Thanks everyone. Cheers, -- jr '' a - Original Message - > From: "Sebastian J. Orsini II via Ou

Re: [outages] Mediacom Outages

2018-10-23 Thread Jay R. Ashworth via Outages
-title.com > > > > Sent from my iPhone*, **Please excuse any abbreviations and auto correct > errors.* > > On Oct 23, 2018, at 5:43 PM, Jay R. Ashworth via Outages < > outages@outages.org> wrote: > > I'll take that, thanks. > > > --- > > A

Re: [outages] Mediacom Outages

2018-10-23 Thread Jay R. Ashworth via Outages
I'll take that, thanks. --- Abhorrent is about 3 notches too strong, but yes, Chris, if there's any way you can avoid dumping all those extra disclaimers on list-mail, that would be greatly appreciated. Remember, you are making 10,000 copies. :-) Cheers, -- jra - Original Message -

[outages] PINCH Re: Slack Outage?

2018-05-21 Thread Jay R. Ashworth via Outages
One meetoo in each direction, folks; this thread's gotten out of hand. Slack is only barely on-topic for the list anyway. Please remember, many list members have the main list on Loud Notify. Cheers, -- jra - Original Message - > From: "Matt Freitag via Outages" > To: "OUTAGES Mailing

Re: [outages] Hughesnet Down

2018-01-02 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Curt Akin via Outages" > > This HughesNet outage is affecting multiple gateways, so is probably a > satellite issue. I would normally expect to have heard of such an event, and I haven't. Downdetector had a peak of problem reports through about 11pm 1/1, ta

[outages] ADMIN: PINCH

2017-11-06 Thread Jay R. Ashworth via Outages
Everyone keeps forgetting this, so I'll say it a bit less politely this time: On the main list: anymore than 1 or 2 "metoo" or "notme" replies TOTAL is too many; please look at the list before you chime in. Jeez, people; I go to lunch and there's 70 new posts... 10,000 Cellphones. That's what y

[outages] REMINDER: DST

2017-11-03 Thread Jay R. Ashworth via Outages
This Sunday, 015959 US local time will be immediately followed by 01, in all states/provinces that observe DST. This means that local apps will experience reverse clock-slew, and that multistate networks will experience multiple incidents of clocks-out-of-phase, *if they work off local time i

[outages] Oops.

2017-10-04 Thread Jay R. Ashworth via Outages
That's what I get for not checking before linking. Thanks to a couple people who pointed out the SSL cert's expired on the wiki; we'll get right on that. :-) Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Thin

Re: [outages] Facebook commenting and Messenger down on Safari since late last evening

2017-10-04 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Charlie via Outages" > This isn't something for this email list. > > Jay Farrell via Outages skrev: (4 oktober 2017 14:23:56 > CEST) >>Facebook is broken in Safari web browser again since late last evening >>(this also happened a month or two ago). Can see

Re: [outages] smugmug.com (on cloudfront) is down

2017-09-16 Thread Jay R. Ashworth via Outages
- Original Message - > From: "Patrick W. Gilmore via Outages" > Couldn’t get to smugmug (where I keep my photos). > > The host “www.smugmug.com” is up, but it redirects to cloudfront: > > TiggerBook-C:~ patrick$ host www.smugmug.com > www.smugmug.com is an alias for dq33tynp

[outages] REMINDER: Irma

2017-09-05 Thread Jay R. Ashworth via Outages
Just as with Harvey and the eclipse, the incoming Hurricane Irma will cause lots of trouble in the Caribbean, in some part of Florida, and possibly further up the coast, depending on the final track of the storm. Since this is prone to cause the noise floor on the mailing list to rise, and since

[outages] REMINDER: ECLIPSE

2017-08-18 Thread Jay R. Ashworth via Outages
Monday, 21 August, for the first time since 1979, a full solar eclipse will pass across the continental United States. This has caused enough excitement among the general public that an estimated 20 million Americans will be along the path of totality, rather than in their normal market areas, and

Re: [outages] thetvdb.com returning 503

2017-08-04 Thread Jay R. Ashworth via Outages
Yeah; that site's probably not big enough to justify posting here. Cheers, -- jra - Original Message - > From: "Andrew Herrington via Outages" > To: outages@outages.org > Sent: Friday, August 4, 2017 9:32:07 AM > Subject: Re: [outages] thetvdb.com returning 503 > Not sure this is outag

  1   2   >