Re: CloudFlare Issues?

All three resolver of the big German hoster Hetzner went offline with 1.1.1.1 and were down another hour after Cloudflare was up again. I hope it was a coincidence and they are not forwarding their requests to 1.1.1.1. https://www.hetzner-status.de/ - vidister > On 18. Jul 2020, at 02:46, John

Re: CloudFlare Issues?

Noticed high latency from some smokeping instances from about 16:10 until 16:35 (central time). One of the worst variances was from ~20ms to upwards of 100ms RTT.

Re: CloudFlare Issues?

We're peered with them and are having issues resolving some domains via Cloudflare right now. On Fri, Jul 17, 2020 at 2:44 PM Aaron C. de Bruyn via NANOG wrote: > More digging shows high latency to CloudFlare DNS servers from Comcast in > Washington and Oregon as well as a few other providers (C

Re: CloudFlare Issues?

Did anyone see any collateral damage from this outside of Cloudflare? Specifically Azure? I manage a very large site in Azure, and at the exact same time of the Cloudflare incident we saw a spike in traffic (like a DDoS or Bot), then followed by unusual hardware resource anomalies. We’re global

Re: CloudFlare Issues?

Once upon a time, Peter Kristolaitis said: > Cloudflare's status page acknowledged a recursive DNS issue as of a > few minutes ago.  Lots of reports of problems on the Outages list > and Reddit. It was not just recursive - authoritative DNS on Cloudflare servers also did not respond. -- Chris Ad

Re: CloudFlare Issues?

The team is working on it. _ *Justin Paine* Head of Trust & Safety PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D 101 Townsend St., San Francisco, CA 94107 On Fri, Jul 17, 2020 at 2

Re: CloudFlare Issues?

> On Jul 17, 2020, at 5:38 PM, Chris Grundemann wrote: > > Looks like there may be something big up (read: down) at CloudFlare, but > their status page is not reporting anything yet. > > Am I crazy? Or just time to give up on the internet for this week? > > You’re not crazy. I’m seeing th

Re: CloudFlare Issues?

CloudFlare updated their status page and confirmed the issue: https://www.cloudflarestatus.com/ -A On Fri, Jul 17, 2020 at 2:33 PM Aaron C. de Bruyn wrote: > More digging shows high latency to CloudFlare DNS servers from Comcast in > Washington and Oregon as well as a few other providers (Char

Re: CloudFlare Issues?

From cloudflarestatus.com Cloudflare Network and Resolver Issues *Investigating* - Cloudflare is investigating issues with Cloudflare Resolver and our edge network in certain locations. Customers using Cloudflare services in certain regio

Re: CloudFlare Issues?

Cloudlflare's status page shows they are investigating an issue. Discord's status page also shows Cloudflare has an issue. Most people aren't making the Cloudflare connection yet and reporting many other services down instead. On Fri, Jul 17, 2020 at 4:40 PM Chris Grundemann wrote: > Looks like

Re: CloudFlare Issues?

Cloudflare's status page acknowledged a recursive DNS issue as of a few minutes ago.  Lots of reports of problems on the Outages list and Reddit. From their status page: *Investigating*- Cloudflare is investigating issues with Cloudflare Resolver and our edge network in certain locations. Cu

RE: CloudFlare Issues?

My sites appear to be normal. Maybe it’s time for happy hour? Thank you, Spencer From: NANOG On Behalf Of Chris Grundemann Sent: Friday, July 17, 2020 4:39 PM To: NANOG list Subject: CloudFlare Issues? CAUTION: This email originated from an external source. Verify the sender before taking an

Re: CloudFlare Issues?

Chris Grundemann wrote on 7/17/2020 2:38 PM: Looks like there may be something big up (read: down) at CloudFlare, but their status page is not reporting anything yet. Am I crazy? Or just time to give up on the internet for this week? -- @ChrisGrundemann http://chrisgrundemann.com Status page

RE: CloudFlare Issues?

https://www.cloudflarestatus.com/ From: NANOG On Behalf Of Chris Grundemann Sent: Friday, July 17, 2020 4:39 PM To: NANOG list Subject: CloudFlare Issues? *External Email: Use Caution* Looks like there may be something big up (read: down) at CloudFlare, but their status page is not reporting

Re: CloudFlare Issues?

I think they were down for about 30 or so minutes, but came back up right about the time you hit the send button --Rob McEwen On 7/17/2020 5:38 PM, Chris Grundemann wrote: Looks like there may be something big up (read: down) at CloudFlare, but their status page is not reporting anything yet.

Re: CloudFlare Issues?

More digging shows high latency to CloudFlare DNS servers from Comcast in Washington and Oregon as well as a few other providers (Charter, ToledoTel), etc... Sites that do resolve using other DNS servers but are hosted on CloudFlare aren't loading. Sites that use CloudFlare for their DNS aren't re

Re: CloudFlare issues?

On 6/Jul/19 23:44, Matt Corallo wrote: > On my test net I take ROA_INVALIDs and convert them to unreachables with > a low preference (ie so that any upstreams taking only the shorter path > will be selected, but so that such packets will never be routed). > > Obviously this isn't a well-supporte

Re: CloudFlare issues?

On 6/Jul/19 22:05, Brett Frankenberger wrote: > These were more-specifics, though. So if you drop all the > more-specifics as failing ROV, then you end up following the valid > shorter prefix to the destination. I can't quite recall which Cloudflare prefixes were impacted. If you have a sniff

Re: CloudFlare issues?

Oops, I mean with a script which removes such routes if there is an encompassing route which a different upstream takes, as obviously the more-specific would otherwise still win. Matt On 7/6/19 5:44 PM, Matt Corallo wrote: > On my test net I take ROA_INVALIDs and convert them to unreachables with

Re: CloudFlare issues?

On my test net I take ROA_INVALIDs and convert them to unreachables with a low preference (ie so that any upstreams taking only the shorter path will be selected, but so that such packets will never be routed). Obviously this isn't a well-supported operation, but I'm curious what people think of s

Re: CloudFlare issues?

On Thu, Jul 04, 2019 at 11:46:05AM +0200, Mark Tinka wrote: > I finally thought about this after I got off my beer high :-). > > Some of our customers complained about losing access to Cloudflare's > resources during the Verizon debacle. Since we are doing ROV and > dropping Invalids, this should

Re: CloudFlare issues?

t. Your message below hints that you may be using RPKI. Are you > doing ROV? (You may be in the “hundreds of others” category.) > > —Sandy > > Begin forwarded message: > > From: Job Snijders > Subject: Re: CloudFlare issues? > Date: July 4, 2019 at 11:33:57 AM EDT >

Re: CloudFlare issues?

Martijn - i3D.net is not in the list Job posted yesterday of RPKI ROV deployment. Your message below hints that you may be using RPKI. Are you doing ROV? (You may be in the “hundreds of others” category.) —Sandy Begin forwarded message: From: Job Snijders Subject: Re: CloudFlare issues

Re: CloudFlare issues?

> Anyway, you can now enjoy https://rpki.net/s/rpki-test even more! :-) my apologies, I fumbled the ball on typing in that URL, I intended to point here: https://www.ripe.net/s/rpki-test

Re: CloudFlare issues?

On 4/Jul/19 20:46, Francois Lecavalier wrote: > It's been close to 3 hours now since I dropped them - radio silence. > > Whoever fears implementing RPKI/ROA/ROV, simply don't. It's very easy to > implement, validate and troubleshoot. Well done! Congrats! Mark.

Re: CloudFlare issues?

On 4/Jul/19 20:46, Francois Lecavalier wrote: > It's been close to 3 hours now since I dropped them - radio silence. > > Whoever fears implementing RPKI/ROA/ROV, simply don't. It's very easy to > implement, validate and troubleshoot. Well done! Congrats! Mark.

Re: CloudFlare issues?

On Thu, Jul 4, 2019 at 8:46 PM Francois Lecavalier wrote: > It's been close to 3 hours now since I dropped them - radio silence. I am going to assume that "radio silence" for you means that your network is fully functional and none of your customers have raised issues! :-) > Whoever fears imple

Re: CloudFlare issues?

Welcome to the club! Get Outlook for Android<https://aka.ms/ghei36> From: Francois Lecavalier Sent: Thursday, July 4, 2019 8:46:46 PM To: Ben Maddison; j...@ntt.net Cc: nanog@nanog.org Subject: RE: CloudFlare issues? >> At this point in time I thi

RE: CloudFlare issues?

- From: Ben Maddison Sent: Thursday, July 4, 2019 11:51 AM To: j...@ntt.net; Francois Lecavalier Cc: nanog@nanog.org Subject: [External] Re: CloudFlare issues? Hi Francois, On Thu, 2019-07-04 at 17:33 +0200, Job Snijders wrote: > Dear Francois, > > On Thu, Jul 04, 2019 at 0

Re: CloudFlare issues?

On 4/Jul/19 17:50, Ben Maddison via NANOG wrote: > We have been dropping Invalids since April, and have had only a > (single-digit) handful of support requests related to those becoming > unreachable. We've had 2 cases where customers could not reach a prefix. Both were mistakes (as we've foun

Re: CloudFlare issues?

On 4/Jul/19 17:33, Job Snijders wrote: > At this point in time I think the ideal deployment model is to perform > the validation within your administrative domain and run your own > validators. In essence, this is also my thought process. I think Cloudflare are very well-intentioned in making

Re: CloudFlare issues?

Francois Lecavalier wrote on 04/07/2019 16:22: My assumption is that 1.Accept valid, 2. Accept unknown, 3. Reject invalid shouldn’t break anything. Accepting valid ROAs is a better idea after checking that the source AS is legitimate from the peer. Nick

Re: CloudFlare issues?

On 4/Jul/19 17:22, Francois Lecavalier wrote: >   > > Following that Verizon debacle I got onboard with ROV, after a couple > research I stopped my choice on the ….drum roll…. CloudFlare GoRTR > (https://github.com/cloudflare/gortr).  If you trust them enough they > provide an updated JSON every

Re: CloudFlare issues?

Hi Francois, On Thu, 2019-07-04 at 17:33 +0200, Job Snijders wrote: > Dear Francois, > > On Thu, Jul 04, 2019 at 03:22:23PM +, Francois Lecavalier wrote: > > > At this point in time I think the ideal deployment model is to > perform > the validation within your administrative domain and run

Re: CloudFlare issues?

Dear Francois, On Thu, Jul 04, 2019 at 03:22:23PM +, Francois Lecavalier wrote: > Following that Verizon debacle I got onboard with ROV, after a couple > research I stopped my choice on the drum roll CloudFlare GoRTR > (https://github.com/cloudflare/gortr). If you trust them enough th

Re: CloudFlare issues?

Hi Mark, Following that Verizon debacle I got onboard with ROV, after a couple research I stopped my choice on the drum roll CloudFlare GoRTR (https://github.com/cloudflare/gortr). If you trust them enough they provide an updated JSON every 15 minutes of the global RIR aggregate. I'll

Re: CloudFlare issues?

So that means it's time for everyone to migrate their ARIN resources to a sane RIR that does allow normal access to and redistribution of its RPKI TAL? ;-) The RPKI TAL problem + an industry-standard IRRDB instead of WHOIS-RWS were both major reasons for us to bring our ARIN IPv4 address space t

Re: CloudFlare issues?

I finally thought about this after I got off my beer high :-). Some of our customers complained about losing access to Cloudflare's resources during the Verizon debacle. Since we are doing ROV and dropping Invalids, this should not have happened, given most of Cloudflare's IPv4 and IPv6 routes are

Re: CloudFlare issues?

>> Respectfully, I believe Cloudflare’s public comments today have been >> a real disservice. This blog post, and your CEO on Twitter today, >> took every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re >> not. > > I presume that seeing a CF blog post isn’t regular for you. :-). never see

Re: CloudFlare issues?

On Tue, Jun 25, 2019 at 7:06 AM Stephen Satchell wrote: > On 6/25/19 2:25 AM, Katie Holly wrote: > > Disclaimer: As much as I dislike Cloudflare (I used to complain about > > them a lot on Twitter), this is something I am absolutely agreeing with > > them. Verizon failed to do the most basic of n

Re: CloudFlare issues?

Hi Stephen, > I used to be a quality control engineer in my career, so I have a > question to ask from the perspective of a QC guy: what is the Best > Practice for minimizing, if not totally preventing, this sort of > problem? Is there a "cookbook" answer to this? > As suggested by Job in the

Re: CloudFlare issues?

On 6/25/19 2:25 AM, Katie Holly wrote: > Disclaimer: As much as I dislike Cloudflare (I used to complain about > them a lot on Twitter), this is something I am absolutely agreeing with > them. Verizon failed to do the most basic of network security, and it > will happen again, and again, and again.

Re: CloudFlare issues?

Verizon Business / Enterprise is the access network, aka 701/2/3. Verizon Media Group is the CDNs/Media side. Digital Media Services ( Edgecast ) , Yahoo, AOL. 15133 / 10310 / 1668. ( The entity formerly named Oath, created when Yahoo was acquired. ) On Tue, Jun 25, 2019 at 06:54 Hank Nussbacher

Re: CloudFlare issues?

On 25/06/2019 08:17, Christopher Morrow wrote: On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher wrote: On 25/06/2019 03:03, Tom Beecher wrote: Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701. My comments are my own opinions only. Respectfully, I believe C

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 09:39:13PM -0400, Ross Tajvar wrote: > A technical one - see below from CF's blog post: > "It is unfortunate that while we tried both e-mail and phone calls to reach > out to Verizon, at the time of writing this article (over 8 hours after the > incident), we have not heard

Re: CloudFlare issues?

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701. My comments are my own opinions only. Disclaimer: As much as I dislike Cloudflare (I used to complain about them a lot on Twitter), this is something I am absolutely agreeing with them. Verizon failed to do

Re: CloudFlare issues?

On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher wrote: > > On 25/06/2019 03:03, Tom Beecher wrote: > > Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do > > not work on 701. My comments are my own opinions only. > > > > Respectfully, I believe Cloudflare’s public comments toda

Re: CloudFlare issues?

On 25/06/2019 03:03, Tom Beecher wrote: Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701.  My comments are my own opinions only. Respectfully, I believe Cloudflare’s public comments today have been a real disservice. This blog post, and your CEO on Twitter t

Re: CloudFlare issues?

> On Jun 24, 2019, at 9:39 PM, Ross Tajvar wrote: > > > On Mon, Jun 24, 2019 at 9:01 PM Jared Mauch wrote: > > > > > On Jun 24, 2019, at 8:50 PM, Ross Tajvar wrote: > > > > > > Maybe I'm in the minority here, but I have higher standards for a T1 than > > > any of the other players involved

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 9:01 PM Jared Mauch wrote: > > > On Jun 24, 2019, at 8:50 PM, Ross Tajvar wrote: > > > > Maybe I'm in the minority here, but I have higher standards for a T1 than any of the other players involved. Clearly several entities failed to do what they should have done, but Veriz

Re: CloudFlare issues?

> On Jun 24, 2019, at 8:50 PM, Ross Tajvar wrote: > > Maybe I'm in the minority here, but I have higher standards for a T1 than any > of the other players involved. Clearly several entities failed to do what > they should have done, but Verizon is not a small or inexperienced operation. > T

Re: CloudFlare issues?

> On Jun 24, 2019, at 8:03 PM, Tom Beecher wrote: > > Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work > on 701. My comments are my own opinions only. > > Respectfully, I believe Cloudflare’s public comments today have been a real > disservice. This blog post,

Re: CloudFlare issues?

Maybe I'm in the minority here, but I have higher standards for a T1 than any of the other players involved. Clearly several entities failed to do what they should have done, but Verizon is not a small or inexperienced operation. Taking 8+ hours to respond to a critical operational problem is what

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 08:03:26PM -0400, Tom Beecher wrote: > > You are 100% right that 701 should have had some sort of protection > mechanism in place to prevent this. But do we know they didn???t? Do we know > it was there and just setup wrong? Did another change at another time break > what w

Re: CloudFlare issues?

--- beec...@beecher.cc wrote: From: Tom Beecher :: Shouldn’t we be working on facts? Nah, this is NANOG... >;-) :: But this industry is one big ass glass house. What’s that :: thing about stones again? We all have broken windows? :) scott

Re: CloudFlare issues?

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701. My comments are my own opinions only. Respectfully, I believe Cloudflare’s public comments today have been a real disservice. This blog post, and your CEO on Twitter today, took every opportunity to say “DAMN TH

Re: CloudFlare issues?

FYI for the group -- we just published this: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ _ *Justin Paine* Director of Trust & Safety PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D 101 Townsend St., San Fran

Re: CloudFlare issues?

On 24/Jun/19 18:09, Pavel Lunin wrote: > > Hehe, I haven't seen this text before. Can't agree more. > > Get your tie back on Job, nobody listened again. > > More seriously, I see no difference between prefix hijacking and the > so called bgp optimisation based on completely fake announces on >

Re: CloudFlare issues?

On 2019-06-24 20:16, Mark Tinka wrote: > > > On 24/Jun/19 16:11, Job Snijders wrote: > >> >> - deploy RPKI based BGP Origin validation (with invalid == reject) >> - apply maximum prefix limits on all EBGP sessions >> - ask your router vendor to comply with RFC 8212 ('default deny') >

Re: CloudFlare issues?

On 24/Jun/19 16:11, Job Snijders wrote: > > - deploy RPKI based BGP Origin validation (with invalid == reject) > - apply maximum prefix limits on all EBGP sessions > - ask your router vendor to comply with RFC 8212 ('default deny') > - turn off your 'BGP optimizers' I cannot ov

Re: CloudFlare issues?

From https://www.cloudflarestatus.com/​: Identified - We have identified a possible route leak impacting some Cloudflare IP ranges and are working with the network involved to resolve this. Jun 24, 11:36 UTC Seeing issues in Australia too for some sites that are routing through Cloudflare. [h

Re: CloudFlare issues?

>I'd like to point everyone to an op-ed I wrote on the topic of "BGP optimizers": >https://seclists.org/nanog/2017/Aug/318 Hehe, I haven't seen this text before. Can't agree more. Get your tie back on Job, nobody listened again. More seriously, I see no difference between prefix hijacking and th

Re: CloudFlare issues?

24.06.19 19:04, Matthew Walster пише: On Mon, 24 Jun 2019, 16:28 Max Tulyev, > wrote: 1. Why Cloudflare did not immediately announced all their address space by /24s? This can put the service up instantly for almost all places Probably RPKI and that being a

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 10:41 AM Filip Hruska wrote: > > Verizon is the one who should've noticed something was amiss and dropped > their customer's BGP session. > They also should have had filters and prefix count limits in place, > which would have prevented this whole disaster. > oddly VZ used

Re: CloudFlare issues?

Verizon is the one who should've noticed something was amiss and dropped their customer's BGP session. They also should have had filters and prefix count limits in place, which would have prevented this whole disaster. As to why any of that didn't happen, who actually knows. Regards, Filip On

Re: CloudFlare issues?

Hi All, here in Ukraine we got an impact as well! Have two questions: 1. Why Cloudflare did not immediately announced all their address space by /24s? This can put the service up instantly for almost all places. 2. Why almost all carriers did not filter the leak on their side, but waited fo

Re: CloudFlare issues?

This is what looked happened: There was a large scale BGP 'leak' incident causing about 20k prefixes for 2400 network (ASNs) to be rerouted through AS396531 (a steel plant) and then on to its transit provider: Verizon (AS701) Start time: 10:34:21 (UTC) End time: 12:37 (UTC) All ASpaths had the fo

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 08:18:27AM -0400, Tom Paseka via NANOG wrote: > a Verizon downstream BGP customer is leaking the full table, and some more > specific from us and many other providers. It appears that one of the implicated ASNs, AS 33154 "DQE Communications LLC" is listed as customer on Noc

Re: CloudFlare issues?

This is my final update, I’m going back to bed, wake me up when the internet is working again. https://news.ycombinator.com/item?id=20262316 —— 1230 UTC update We are working with networks around the world and are observing network routes for Google and AWS being leaked at well. On Mon, Jun 24,

Re: CloudFlare issues?

*1204 UTC update* This leak is wider spread that just Cloudflare. *1208 UTC update* Amazon Web Services now reporting external networking problem On Mon, Jun 24, 2019 at 05:18 Tom Paseka wrote: > a Verizon downstream BGP customer is leaking the full table, and some more > specific from us and m

Re: CloudFlare issues?

a Verizon downstream BGP customer is leaking the full table, and some more specific from us and many other providers. On Mon, Jun 24, 2019 at 7:56 AM Robbie Trencheny wrote: > *1147 UTC update* Staring at internal graphs looks like global traffic is > now at 97% of expected so impact lessening.

Re: CloudFlare issues?

*1147 UTC update* Staring at internal graphs looks like global traffic is now at 97% of expected so impact lessening. On Mon, Jun 24, 2019 at 04:51 Dovid Bender wrote: > We are seeing issues as well getting to HE. The traffic is going via Alter. > > > > On Mon, Jun 24, 2019 at 7:48 AM Robbie Tre

Re: CloudFlare issues?

We are seeing issues as well getting to HE. The traffic is going via Alter. On Mon, Jun 24, 2019 at 7:48 AM Robbie Trencheny wrote: > From John Graham-Cumming, CTO of Cloudflare, on Hacker News right now: > > This appears to be a routing problem with Level3. All our systems are > running norma

Re: CloudFlare issues?

>From John Graham-Cumming, CTO of Cloudflare, on Hacker News right now: This appears to be a routing problem with Level3. All our systems are running normally but traffic isn't getting to us for a portion of our domains. 1128 UTC update Looks like we're dealing with a route leak and we're talking

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 02:03:47PM +0300, Antonios Chariton wrote: > Yes, traffic from Greek networks is routed through NYC (alter.net > ), and previously it had a 60% packet loss. Now it???s > still via NYC, but no packet loss. This happens in GR-IX Athens, not GR-IX > Thessa

Re: CloudFlare issues?

Yes, traffic from Greek networks is routed through NYC (alter.net ), and previously it had a 60% packet loss. Now it’s still via NYC, but no packet loss. This happens in GR-IX Athens, not GR-IX Thessaloniki, but the problem definitely exists. Antonis > On 24 Jun 2019, at 13