Re: Help Us Measure Fixed 5G Networks
A good place to ask is DSLReports or Reddit. On both sites, many people may be interested in doing this research. -Neel On 2023-06-26 09:38, Nick Feamster wrote: Hello, We are working on a project to measure the performance of Fixed 5G wireless networks. If you or anyone you know are a subscriber of: * Starlink * Verizon * T-Mobile you are eligible to help us with this study. Participation is simple: We send a Raspberry Pi with some pre-installed measurement software (Netrics), you plug it into the wall and your cable modem, and we give you a performance dashboard with your network performance, including comparisons to other networks. We offer a small participation incentive, as well. We’d ask for participation for 1-2 months, although you are welcome to continue your participation longer. See the form below for more information, including how to participate: https://uchicago.co1.qualtrics.com/jfe/form/SV_eCFrZhRhNphkVGm Thanks, Nick Feamster and Francesco Bronzino
IPv4 Subnet 23.151.232.0/24 blackholed?
Hi, I recently got the IPv4 allocation 23.151.232.0/24 from ARIN. I also had my hosting company ReliableSite announce it to the internet. Right now, I can only access networks that peer with ReliableSite via internet exchanges, such as Google, CloudFlare, OVH, Hurricane Electric, et al. It seems the Tier 1 ISPs (e.g. Lumen, Cogent, AT, et al.) are blackholing the IPv4 subnet 23.151.232.0/24. Could someone who works at a Tier 1 NOC please check and remove the blackhole if any exists? Normally when ReliableSite announced my prior (then-leased) IPv4 space it gets propagated via BGP almost immediately. This time it's not going through at all. Best, Neel Chauhan
GTT blocking IPv4 address 128.31.0.39
Hi, I am a customer of ReliableSite in their New Jersey location, and RS uses GTT as a transit ISP, along with Tata and Comcast. GTT appears to be blocking the IPv4 address 128.31.0.39, and RS' BGP uses GTT for 128.31.0.39. neel@t1:~ % traceroute 128.31.0.39 traceroute to 128.31.0.39 (128.31.0.39), 64 hops max, 40 byte packets 1 45.150.XXX.1 (45.150.XXX.1) 4.828 ms 4.557 ms 5.916 ms 2 * * * ^C neel@t1:~ % Hop #2 which is generally the transit provider, GTT (which handles this route). Note: 45.150.XXX.1 is because it's a subnet I brought in, this is the only ReliableSite hop. The 128.31.0.0/24 doesn't appear to be blocked as a whole, only that 128.31.0.39. See below: neel@t1:~ % traceroute 128.31.0.1 traceroute to 128.31.0.1 (128.31.0.1), 64 hops max, 40 byte packets 1 45.150.XXX.1 (45.150.XXX.1) 0.241 ms 0.220 ms 9.362 ms 2 ae9-300.cr2-nyc4.ip4.gtt.net (209.120.147.121) 1.605 ms 0.853 ms 1.173 ms 3 ae3.cr1-nyc2.ip4.gtt.net (89.149.129.194) 5.488 ms 6.471 ms 1.451 ms 4 be3088.ccr31.jfk04.atlas.cogentco.com (154.54.11.57) 1.604 ms 1.726 ms * 5 be3363.ccr42.jfk02.atlas.cogentco.com (154.54.3.125) 1.802 ms 1.771 ms 1.708 ms 6 be3472.ccr32.bos01.atlas.cogentco.com (154.54.46.33) 7.082 ms 7.268 ms 7.249 ms 7 38.104.186.186 (38.104.186.186) 7.017 ms 7.247 ms 6.987 ms 8 dmz-rtr-1-external-rtr-3.mit.edu (18.0.161.13) 7.010 ms 7.001 ms 6.996 ms 9 dmz-rtr-2-dmz-rtr-1-2.mit.edu (18.0.162.6) 7.033 ms 7.294 ms dmz-rtr-2-dmz-rtr-1-1.mit.edu (18.0.161.6) 7.073 ms 10 guest.default.csail.mit.edu (128.31.0.1) 9.011 ms 7.484 ms 7.551 ms neel@t1:~ % As you can see here, GTT handles other 128.31.0.39 IPs fine as seen in hop #2. ReliableSite says they don't block the IP address, but I don't have any contact at GTT or MIT. My home ISP, Lumen/CenturyLink/Level 3 does not block 128.31.0.39. 128.31.0.39 is a Tor directory authority IP, which is usually a phonebook of Tor relays. There are 9 in the world and the other 8 are unblocked from ReliableSite. Yes, I know Tor is all this 'bad stuff' but the reality is that 99% of Tor users use it like a VPN, speaking as a Tor exit operator and code contributor myself. Exit abuse complaints were super common 5-8 years ago but are now super rare. If someone works at GTT, can 128.31.0.39 be unblocked? Best, -Neel --- https://www.neelc.org
Re: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi, I have taught of an (hackish) workaround for now. Enable my Tor relays, but at the same time switch my non-Tor traffic to Verizon "LTE Home". Then hope my neighbors have service calls with CenturyLink which forces them to fix the issue (a tech told me about "capacity issues"). Monitor the CL connection every day, and if or when CenturyLink fixes the issue, cancel Verizon and enjoy. I could get Xfinity Prepaid for much cheaper, but since the Coax drop on our house is cut and we have no Coax outlets, the install would be hairy and long. CenturyLink had an advantage here since while the home was being flipped CL upgraded the street to fiber. The copper drop is still there and attached (Bell System 305A2 anyone?), but will probably never be used again. -Neel On 2021-11-02 07:28, Neel Chauhan wrote: I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic. The issue occurs even if I'm using 50 Mbps out of my 940. It may be bufferbloat on CL's side but they keep denying the issue. I guess I'll have to break the bank and get Comcast Gigabit Pro. CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice. -Neel On 2021-11-01 20:52, Ryan Hamel wrote: Neel, Sounds like buffer bloat. Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic. Ryan -Original Message- From: NANOG On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA) Hi NANOG Mailing List, I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story. If I attempt to run certain applications that use 1000, or 1 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run). On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS). The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members. On an AT forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio n The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT's 802.1X. I know AT and CenturyLink don't compete, but if I had to choose between AT Fiber and CenturyLink, I'll take AT in a heartbeat, no ifs, no buts, even if I have to use AT's crappy router instead of my OPNsense box. Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle? I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there. I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink. Best, Neel Chauhan
Re: CenturyLink Fiber Latency Issues (Seattle, WA)
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic. The issue occurs even if I'm using 50 Mbps out of my 940. It may be bufferbloat on CL's side but they keep denying the issue. I guess I'll have to break the bank and get Comcast Gigabit Pro. CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice. -Neel On 2021-11-01 20:52, Ryan Hamel wrote: Neel, Sounds like buffer bloat. Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic. Ryan -Original Message- From: NANOG On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA) Hi NANOG Mailing List, I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story. If I attempt to run certain applications that use 1000, or 1 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run). On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS). The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members. On an AT forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio n The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT's 802.1X. I know AT and CenturyLink don't compete, but if I had to choose between AT Fiber and CenturyLink, I'll take AT in a heartbeat, no ifs, no buts, even if I have to use AT's crappy router instead of my OPNsense box. Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle? I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there. I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink. Best, Neel Chauhan
CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List, I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story. If I attempt to run certain applications that use 1000, or 1 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run). On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS). The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members. On an AT forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturation The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT's 802.1X. I know AT and CenturyLink don't compete, but if I had to choose between AT Fiber and CenturyLink, I'll take AT in a heartbeat, no ifs, no buts, even if I have to use AT's crappy router instead of my OPNsense box. Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle? I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there. I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink. Best, Neel Chauhan
Throughput issues on Wave G/CondoInternet (Redmond, WA)
Hi NANOG mailing list,
I have Wave G "Gigabit" service in Redmond, WA, and download speeds have
been slow. Wave G is Wave Broadband's MDU-focused Gigabit offering, and
was formerly CondoInternet, although I joined only this year on a Wave
G-branded service.
To Seattle, I cap at about 600-750 Mbps download while I get 900 Mbps
uploads. To servers far away, it does gown to 10-30 Mbps while I can
upload at 200-500 Mbps. The downloads speeds are far lower than what's
normal from TCP (I **do** know higher latency == slower downloads).
The issue appears to be the window size being smaller on downloads on
the routers on Wave G's side and that limits download throughput.
This is a Wireshark screenshot of a download test:
https://i.imgur.com/2gXbNSZ.png
The green line is the window size, and the blue line is the outstanding
bytes.
In comparison, this is a Wireshark screenshot of an upload test:
https://i.imgur.com/WOGvpWx.png
The download test shows a small window size and spikes in the
outstanding bytes and this gets reset very often after the first
increase instead of gradually using the pipe's full capacity and then
resetting, showing the window size can't get higher.
The upload test shows a larger window size and a sawtooth shape of the
outstanding bytes which is what should normally happen.
This happens on both FreeBSD 13-CURRENT and Windows 10, both hardwired
and Wi-Fi. My work Windows 10 PC via RDP and personal FreeBSD VPS
servers have no issues whatsoever to the same exact servers, both being
in the Seattle metro.
"Tech Support" hasn't been helpful since they insist on a tech visit
which I believe isn't needed ("Wave G" isn't Wave's DOCSIS offering, no
HFC issues here), or blame the issue on something outside their network
(these issues **don't** exist outside Wave G). Ziply/Frontier and
CenturyLink aren't available here, and why would I go to Comcast's
overpriced, asymmetrical, capped garbage?
If someone works for Wave Broadband, or more specifically on the Wave G
service, please shoot me an private email so we can get this resolved
(and I can share my service details and pcap file if necessary in the
said private email).
-Neel Chauhan
===
https://www.neelc.org/
Verizon/UUNET AS701 blocking Tor "directory" server (IPv4 86.59.21.38)
Hi nanog mailing list, Keep in mind that I am not a practicing network engineer, although I do have interest and knowledge on networking topics. I do not work for Verizon. I subscribe to Verizon FiOS, but not Verizon Wireless or Verizon's enterprise services. The Tor "directory" server with the IPv4 address 86.59.21.38 has been blocked by Verizon's AS701 backbone for a few months now. AS701 provides Internet connectivity to Verizon FiOS and Wireless. The design of Tor is that even though anyone can set up a "relay", there are a few central directory servers which clients go to first to get a list of relay servers and build a circuit (which is a path of three relays to reach a destination). A more descriptive overview of Tor is available here: https://www.torproject.org/about/overview.html.en . While I can still access other Tor directory servers from Verizon FiOS as running Tor as a client or relay does not require every directory server be unblocked, blocking one of them could possibly mean breaking some part of the Internet for a Verizon customer. A traceroute to 86.59.21.38 from FiOS shows that I can get through verizon-gni.net which is Verizon's internal FiOS network, but not ALTER.NET, which is Verizon's UUNet backbone: neel@xb2:~ % traceroute 86.59.21.38 traceroute to 86.59.21.38 (86.59.21.38), 64 hops max, 40 byte packets 1 unknown (192.168.1.1) 1.128 ms 0.780 ms 0.613 ms 2 lo0-100.NYCMNY-VFTTP-401.verizon-gni.net (173.68.77.1) 1.001 ms 3.632 ms 0.900 ms 3 B3401.NYCMNY-LCR-22.verizon-gni.net (100.41.137.96) 2.291 ms B3401.NYCMNY-LCR-21.verizon-gni.net (100.41.137.94) 3.172 ms 4.046 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * ^C neel@xb2:~ % In a normal traceroute, I would see ALTER.NET on hop 5. Also, this filtering is not a subnet filtering. A traceroute to 86.59.21.1 works: neel@xb2:~ % traceroute 86.59.21.1 traceroute to 86.59.21.1 (86.59.21.1), 64 hops max, 40 byte packets 1 unknown (192.168.1.1) 0.863 ms 0.757 ms 0.579 ms 2 lo0-100.NYCMNY-VFTTP-401.verizon-gni.net (173.68.77.1) 1.010 ms 1.545 ms 1.034 ms 3 B3401.NYCMNY-LCR-22.verizon-gni.net (100.41.137.96) 3.616 ms B3401.NYCMNY-LCR-21.verizon-gni.net (100.41.137.94) 5.696 ms 10.062 ms 4 * * * 5 0.et-5-1-5.BR3.NYC4.ALTER.NET (140.222.2.127) 3.492 ms 3.506 ms 2.996 ms 6 204.255.168.118 (204.255.168.118) 8.462 ms 7.479 ms 7.252 ms 7 144.232.4.84 (144.232.4.84) 5.041 ms 4.688 ms sl-crs3-lon-0-6-3-0.sprintlink.net (144.232.9.165) 71.865 ms 8 sl-crs2-lon-0-0-3-0.sprintlink.net (213.206.128.181) 72.214 ms 73.579 ms 72.339 ms 9 213.206.129.142 (213.206.129.142) 81.390 ms sl-crs4-ams-0-7-0-3.sprintlink.net (213.206.129.139) 85.854 ms 93.238 ms 10 217.149.47.46 (217.149.47.46) 79.004 ms 85.669 ms 79.392 ms 11 ams5-core-1.bundle-ether1.tele2.net (130.244.82.54) 86.507 ms 78.374 ms 77.740 ms 12 ams-core-2.bundle-ether9.tele2.net (130.244.82.57) 79.642 ms 77.926 ms 81.515 ms 13 wen3-core-2.bundle-ether15.tele2.net (130.244.71.47) 105.400 ms 105.089 ms 109.751 ms 14 tele2at-bundle2-vie3.net.uta.at (212.152.189.65) 122.716 ms 110.820 ms 114.354 ms 15 86.59.21.1 (86.59.21.1) 106.389 ms * 105.379 ms neel@xb2:~ % I had posted this finding on Tor's mailing list (https://lists.torproject.org/pipermail/tor-relays/2018-May/015218.html). I am posting here as (I believe) Verizon NOC people are more likely to read NANOG mailing lists than Tor mailing lists, although this post is modified from the original because not all network engineers may know how Tor works. From Tor developer Roger Dingledine (at the Tor mailing list), the reason why Verizon blocked 86.59.21.38 in the first place is probably the WannaCry ransomware, and the VZ NOC didn't realize it was a Tor IP address (or how Tor works), and then whoever did this block forgot about it and moved on. I can understand that you all may not know how Tor works either, so I included an overview link above. It could also be possible that it's the NN repeal (but less likely since it is on the level of UUNET not FiOS). I also contacted the operator of 86.59.21.38 as well as Verizon FiOS support, and neither were of much help (the former is obvious as he's Austrian). Well, thank you for reading. Best, Neel Chauhan === https://www.neelc.org/
