RE: CenturyLink Fiber Latency Issues (Seattle, WA)

[Ryan Hamel]

Neel,

Sounds like buffer bloat.

Run a speed test, whatever is your maximum for your download and upload take
10% away from it, and setup traffic shaping in OPNsense
(https://docs.opnsense.org/manual/shaping.html) with those values. If the
issue goes away, then you're exceeding the buffer of CenturyLink's device
with the bursts of traffic.

Ryan

-Original Message-
From: NANOG  On Behalf Of Neel
Chauhan
Sent: Monday, November 1, 2021 6:44 PM
To: nanog@nanog.org
Subject: CenturyLink Fiber Latency Issues (Seattle, WA)

Hi NANOG Mailing List,

I don't know if any of you work at CenturyLink/Lumen, very less on their
Fiber network in Seattle, WA. However, here's my story.

If I attempt to run certain applications that use 1000, or 1 TCP
connections, I get latency spikes. It is based on how many connections, but
also how much bandwidth is used. This means certain things like Tor relays
are off limits to me (which I wish to run).

On an idle connection, the PingPlotter outputs look like this: 
https://centurylinklatencyissues.com/image-000.png

If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter
looks like this: 
https://centurylinklatencyissues.com/image-002.png

Getting support, or even executive contacts to admit the issue hasn't
worked. They all love to blame my equipment or applications, when CL routers
also show the issue when I run the same things whereas my same exact
OPNsense box on Google Fiber Webpass running Tor at another address had no
issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just
fine (from a VPS).

The most competent person I dealt with was actually one tech. He told me
there was "capacity issues" in our neighborhood, and that's the reason for
the issues. However, nothing was done about it afterwards, I'm guessing
since I turned off my Tor relay after the visit to avoid complaints from
family members.

On an AT forum, people have said GPON gives latency spikes/packet loss on
congestion: 
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
n

The capacity managers in Seattle are literally dragging their feet: it's
100x worse than AT's 802.1X. I know AT and CenturyLink don't compete,
but if I had to choose between AT Fiber and CenturyLink, I'll take AT in
a heartbeat, no ifs, no buts, even if I have to use AT's crappy router
instead of my OPNsense box.

Going back, do any of you who work at CenturyLink/Lumen can get me to the
right people, hopefully the capacity managers in Seattle?

I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo
for "Gigabit Pro" with a 2-year contract and a steep install fee. I am
seriously considering Gigabit Pro even if it breaks the bank, but hope I
won't have to go there.

I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads
when I need it is the sweet spot for me (even without Tor) which CL GPON
should easily handle without a sweat. I also don't exactly
**trust** Comcast, they're a horrible company in many metrics, but in some
ways Comcast is more competent than CenturyLink.

Best,

Neel Chauhan

CenturyLink Fiber Latency Issues (Seattle, WA)

[Neel Chauhan]

Hi NANOG Mailing List,

I don't know if any of you work at CenturyLink/Lumen, very less on their 
Fiber network in Seattle, WA. However, here's my story.


If I attempt to run certain applications that use 1000, or 1 TCP 
connections, I get latency spikes. It is based on how many connections, 
but also how much bandwidth is used. This means certain things like Tor 
relays are off limits to me (which I wish to run).


On an idle connection, the PingPlotter outputs look like this: 
https://centurylinklatencyissues.com/image-000.png


If I attempt to run BitTorrent with 1000 connections in Deluge, 
PingPlotter looks like this: 
https://centurylinklatencyissues.com/image-002.png


Getting support, or even executive contacts to admit the issue hasn't 
worked. They all love to blame my equipment or applications, when CL 
routers also show the issue when I run the same things whereas my same 
exact OPNsense box on Google Fiber Webpass running Tor at another 
address had no issues whatsoever, and I can ping other Tor relays on 
CenturyLink AS209 just fine (from a VPS).


The most competent person I dealt with was actually one tech. He told me 
there was "capacity issues" in our neighborhood, and that's the reason 
for the issues. However, nothing was done about it afterwards, I'm 
guessing since I turned off my Tor relay after the visit to avoid 
complaints from family members.


On an AT forum, people have said GPON gives latency spikes/packet loss 
on congestion: 
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturation


The capacity managers in Seattle are literally dragging their feet: it's 
100x worse than AT's 802.1X. I know AT and CenturyLink don't 
compete, but if I had to choose between AT Fiber and CenturyLink, I'll 
take AT in a heartbeat, no ifs, no buts, even if I have to use AT's 
crappy router instead of my OPNsense box.


Going back, do any of you who work at CenturyLink/Lumen can get me to 
the right people, hopefully the capacity managers in Seattle?


I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) 
$329/mo for "Gigabit Pro" with a 2-year contract and a steep install 
fee. I am seriously considering Gigabit Pro even if it breaks the bank, 
but hope I won't have to go there.


I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps 
uploads when I need it is the sweet spot for me (even without Tor) which 
CL GPON should easily handle without a sweat. I also don't exactly 
**trust** Comcast, they're a horrible company in many metrics, but in 
some ways Comcast is more competent than CenturyLink.


Best,

Neel Chauhan

Re: Need for historical prefix blacklist (`rogue' prefixes) information

[David Conrad]

I’m a little confused.  I thought the concern was about decrypting 
intentionally mis-routed traffic, not a suggestion that ROV uses encryption…

Regards,
-drc

> On Oct 30, 2021, at 5:57 PM, J. Hellenthal via NANOG  wrote:
> 
> He answered it completely. "You" worried about interception of RPKI exchange 
> over the wire are failing to see that there is nothing there important to 
> decrypt because the encryption in the transmission is not there !
> 
> And yet you've failed to even follow up to his question... "What's your point 
> regarding your message? ROV does not use (nor needs) encryption."
> 
> So maybe you could give some context on that so someone can steer you out of 
> the wrong direction.
> 
> -- 
>  J. Hellenthal
> 
> The fact that there's a highway to Hell but only a stairway to Heaven says a 
> lot about anticipated traffic volume.
> 
>> On Oct 30, 2021, at 10:31, A Crisan  wrote:
>> 
>> 
>> Hi Matthew, 
>> 
>> Quantum computing exists as POCs, IBM being one of those advertising them 
>> and announced to extend their project. There are others on the market, 
>> Amazon advertised quantum computing as a service back in 2019: 
>> https://www.theverge.com/2019/12/2/20992602/amazon-is-now-offering-quantum-computing-as-a-service
>>  
>> .
>>  The bottle neck of the current technology is scalability: we will not see 
>> QC as personal computing level just yet (to go in more detail, current 
>> technologies work at cryogenic temperatures, thus they are hyper expensive 
>> and not really scalable), but they exist and one could be imagine they 
>> are/will be used for various tasks.
>> 
>> On the other hand, you've actually commented every word of my mail, minus 
>> the stated question. Thanks. 
>> 
>> Best Regards, 
>> Dora Crisan 
>> 
>> 
>> 
>>  
>> 
>> On Fri, Oct 29, 2021 at 8:10 PM Matthew Walster > > wrote:
>> 
>> 
>> On Fri, 29 Oct 2021, 15:55 A Crisan, > > wrote:
>> Hi Matthew,
>> I was reading the above exchange, and I do have a question linked to your 
>> last affirmation. To give you some context, the last 2021 ENISA report seem 
>> to suggest that internet traffic is "casually registered" by X actors to 
>> apply post Retrospective decryption (excerpt below). This would be at odds 
>> with your (deescalating) affirmation that hijacks are non-malicious and they 
>> are de-peered quickly, unless you pinpoint complete flux arrest only. Are 
>> there any reportings/indicators... that look into internet flux constant 
>> monitoring capabilities/capacities? Thanks.
>> 
>> RPKI uses authentication not confidentiality. There is no encryption taking 
>> place, other than the signatures on the certificates etc.
>> 
>> Excerpt from the introduction: "What makes matters worse is that any cipher 
>> text intercepted by an attacker today can be decrypted by the attacker as 
>> soon as he has access to a large quantum computer (Retrospective decryption).
>> 
>> Which do not exist (yet).
>> 
>> Analysis of Advanced Persistent Threats (APT) and Nation State capabilities,
>> 
>> Buzzwords.
>> 
>> along with whistle blowers’ revelations
>>  have shown that threat actors can and are casually recording all Internet 
>> traffic in their data centers
>> 
>> No they're not. It's just not possible or indeed necessary to duplicate 
>> everything at large scale. Perhaps with a large amount of filtering, certain 
>> flows would be captured, but in the days of pervasive TLS, this seems less 
>> and less worthwhile.
>> 
>>  and that they select encrypted traffic as interesting and worth 
>> storing.This means that any data encrypted using any of the standard 
>> public-key systems today will need to be considered compromised once a 
>> quantum computer exists and there is no way to protect it retroactively, 
>> because a copy of the ciphertexts in the hands of the attacker. This means 
>> that data that needs to remain confidential after the arrival of quantum 
>> computers need to be encrypted with alternative means"
>> 
>> None of this is relevant to RPKI (ROV) at all. In fact, it reads like the 
>> fevered dreams of a cyber security research student. What's your point 
>> regarding your message? ROV does not use (nor needs) encryption.
>> 
>> M
>> 

South Korea KT to pay USD$33.97 million for network outage

[Sean Donelan]

https://www.koreatimes.co.kr/www/tech/2021/11/133_318025.html
KT to pay W40 bil. in compensation for network outage

KT will pay out up to 40 billion won ($33.97 million) in compensation to 
customers of its wired and wireless services, which underwent nationwide 
disruptions Oct. 25, the company said.


[...]
KT said it will pay compensation for 10 times the disruption time of 89 
minutes...


[...]
NTT Docomo, AT, Verizon and T-Mobile caused disruptions in services for 
subscribers (customers) in recent years, but none offered compensation.


"We are offering the greatest level of compensation compared to both local 
and foreign companies in the same industry," a KT official said.


[...]
The Ministry of Science and ICT said the mistake of an employee of a KT 
subcontractor caused the nationwide disruption.

Re: possible rsync validation dos vuln

[Giovane C. M. Moura via NANOG]

Good news: the disclosure has been postponed


Quoting from:

https://english.ncsc.nl/latest/news/2021/october/29/upcoming-announcement-of-rpki-cvd-procedure

Update 31 October: Talks have resumed, disclosure is postponed.

Since 30 October, constructive conversation is fortunately taking place
with the parties involved. As such, the NCSC will not publish details
about this vulnerability on Monday 1 November, as previously announced,
but at a later moment (in agreement with the parties involved).


/giovane

On 10/29/21 3:03 AM, Randy Bush wrote:
> received this vuln notice four days before these children intend to
> disclose.  so you can guess how inclined to embargo.
> 
> randy
> 
> 
> From: Koen van Hove 
> Subject: CVD: Vulnerabilities in RPKI Validators
> To: ra...@psg.com, s...@hactrn.net
> Cc: c...@ncsc.nl
> Date: Wed, 27 Oct 2021 14:59:21 -0700
> 
> Dear Randy Bush and Rob Austein,
> 
> Apologies, this email was previously sent to the wrong email address.
> 
> On behalf of the University of Twente and the National Cyber Security
> Centre of the Netherlands (NCSC-NL) we want to notify you of a Coordinated
> Vulnerability Disclosure for RPKI vulnerabilities that also impact rcynic
> developed by Dragon Research Labs.
> 
> The vulnerabilities were discovered by scientific research on the
> implementation of RPKI validators.
> Together with you, the NCSC-NL, the University of Twente, and multiple
> other parties, we would like to come to a timely solution before the
> results of this research will be made public. More information about
> Coordinated Vulnerability Disclosure can be found here [1].
> 
> The vulnerabilities are classified as a denial of service vulnerability and
> impact multiple implementations of RPKI validators including rcynic. Since
> RPKI is of international interest we hope that you will work together with
> us on this CVD.
> 
> The goal is to have fixes available before 1 November which will also be
> the date that the results of this research will become public. Before 1
> November the information in the CVD, or the fact that a CVD is taking
> place, is to be kept strictly confidential. The fixes are to be released
> collectively on 1 November.
> 
> Please let us know whether you agree to these terms, and want to
> participate in this CVD. If so, we will send you the details. We hope to
> hear from you.
> 
> If there are any further questions, please let us know.
> 
> Yours sincerely,
> 
> Koen van Hove
> University of Twente
> 
> [1] https://english.ncsc.nl/contact/reporting-a-vulnerability-cvd
> 
> - --
> Koen van Hove
> -BEGIN PGP SIGNATURE-
> Version: FlowCrypt Email Encryption 8.1.5
> Comment: Seamlessly send and receive encrypted email
> 
> wsDzBAEBCgAGBQJhecu4ACEJEPnqm/++VTh9FiEE5Q3GCKqW0RQyUpA/+eqb
> /75VOH1CjwwAq8Hd0psDhfj6mL4X9ybLGogONpzFKYp9Okv9/CKzQvG4AkLR
> Cvrz3vHlQRKJP8I2PYSLZvtG9D/HXjjKcU+m24jjl2qbKKuSwprqQhLAqabN
> Md+RZFjQGve5Z4vtJsfhXKc4PhaAzMujVc4Mh5Mdbs4sFEdrub1hSnYKlcQV
> PvS/O9SpCYU0E0IC1I455HXxSXUtme+KHtzbGIWQe/mz4KpnZD2Me/Cr1LvG
> Od9izri0Qx5vF+kdpR51PEiwHgN+QkmnUP6Gkrca8TSC2x3ta9B1/ZprdCoZ
> ZYQ7QUFUAkfV+tKCMaBECNOrnDjw8E9GonvzmqpDHBtKBZ3LaxjZX/sxuuTC
> +Ele5nVeWW0ZFqrbanbPy9y1q04tFQd8ewdSN40iXdTj7Ha8GadUhcdSLWqJ
> cLmf71qUAvdwpp0Bt1nhExpU/bEtAaxfnEcTRDX43yUkZXSqV5BxYEyneSLj
> IvFV9AUi56Cx45ESkGRR1ASuCzoc8FCjRH7KOWnaL3fl
> =YQZI
> -END PGP SIGNATURE-
>