Centurylink\Lumen
There's a nearby Lumen route (originally Digital Teleport) where they have built their route to someone else's network where they've built a POP between their own construction and that third party dark fiber provider. That dark fiber provider is friendly. Knowing what they've built out in other places (compared to the number of strands available on the route), I was looking to utilize that lateral to get into the Centurylink POP from the dark fiber network. They've more or less said it doesn't exist and that it'd be $100k for the privilege of paying them more than I'm paying anyone else for a wave. I know it exists because they have no other way out. I'm hoping to find someone in Centurylink\Lumen with a clue that can look up a POP, look at what cables are actually there, strand counts, how big their IRU is, identify excess assets and let me use them to overpay them for service. Why I'm so specifically wanting Lumen is there are very limited options on the route I'm asking for. I'm already doing business with the others, so I'd like some entity diversity. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
RE: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around?
I have multiple clients with the same problem involving tunnels terminated on Lumen to terminations on Comcast. The problems started on Monday morning, 10/11/21, all in the Denver metro area. I was able to change a tunnel from IPSec-ESP to IPSec-HA and the problem went away and then changing back to ESP and the loss came back. We are also seeing problem with IPSec remote VPN clients from Comcast networks terminated to firewalls with Lumen interfaces having drops and the tunnel is unusable. We have opened tickets at Lumen, but they get closed because a traceroute shows no drops and they say the problem is with the "application". John Gawf john.g...@sajens.com<mailto:john.g...@sajens.com> Saje Network Systems 3775 Iris Ave Suite 2C Boulder CO 80301 http://sajens.com<http://sajens.com/> 303.909.6247 From: NANOG On Behalf Of Pennington, Scott Sent: Friday, October 15, 2021 10:48 AM To: nanog@nanog.org Subject: Re: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around? We have also seen the same behavior of intermittent customer complaint followed by issue resolving spontaneously. Our end customer has a tunnel to a supplier on Comcast with a return path via Lumen. The first ticket was opened on 9/23, has cleared and returned a few times. End customer worked with Cisco TAC and symptom is up to 30% ESP packet drop for periods from 5 minutes to 1 hour when the problem is active. From: NANOG mailto:nanog-bounces+scott.pennington=cinbell@nanog.org>> on behalf of Mike Lewinski via NANOG mailto:nanog@nanog.org>> Sent: Thursday, October 14, 2021 2:30 PM To: Brie mailto:br...@2mbit.com>>; nanog@nanog.org<mailto:nanog@nanog.org> mailto:nanog@nanog.org>> Subject: RE: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around? I can confirm this issue exists at several sites in the Denver area with this same IPSEC issue, all routing between Level3/Lumen and Comcast. I was told by one customer that it resolved late yesterday afternoon but I haven't been able to confirm that. Mike -Original Message- From: NANOG mailto:nanog-bounces+mlewinski=massivenetworks@nanog.org>> On Behalf Of Brie Sent: Thursday, October 14, 2021 10:43 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around? Hi all, So, having a... frustrating issue going on. Long wall of text ahead as I explain. 1 x CenturyLink/Lumen fiber in Boise 1 x CenturyLink/Lumen fiber in Cheyenne 1 x Comcast biz fiber in Denver IPsec VPN tunnels between all three sites, w/ OSPF for routing failover (which unfortunately doesn't help in this situation). Two days ago, Cheyenne to Denver (.196) traffic (both tcp and udp) were an issue initially. Failed over to routing Cheyenne VPN through Boise while we opened ticket with CL. Yesterday, Boise to Denver (.196) traffic started having exact same issue. Tests from another CL fiber in Boise (my own circuit, with legacy IP space and BGP) to Denver (.196) did not show same issues. Path appeared clean. Traceroutes from Office Boise to Denver (.196) had a noticeable difference from Personal Boise to Denver (.196): Office Boise -> Denver (.196) -- 3: sea-edge-15.inet.qwest.net 4: lag-4.ear3.Seattle1.Level3.net 5: ae-2-52.ear2.seattle1.level3.net <-- This hop 6: be-203-pe01.seattle.wa.ibone.comcast.net Personal Boise -> Denver (.196) -- 4: sea-edge-15.inet.qwest.net 5: lag-25.ear2.Seattle1.Level3.net 6: be-203-pe01.seattle.wa.ibone.comcast.net On a whim, tracerouted to another Denver router IP address (.199, IP alias on same interface) from Boise Office, and traceroute matched the traceroute from Personal Boise to Denver (.196) traceroute. No packet loss. Swapped VPN tunnels over to using another ip on same router (.199), same interface physical and logical, in Denver, and VPN was working again normally. This morning though, Cheyenne to Denver (.199) is having problems, while Boise to Denver (.199) isn't (for now). Already spent most of last night working with my partner in Denver replacing nearly everything on the Denver side with no change. Tests from the router above the main Denver VPN endpoint (.196) do not show any kind of issues or packet loss to it, so it doesn't appear the problem is inside of our network. I'm not inclined to think this is a Comcast issue, but I can't be sure. This sounds almost like a load balancing hashing issue, with one link in the bond group having issues, somewhere in one of our upstream's networks. We'll be opening a ticket in a bit through normal channels with CenturyLink/Lumen, but we're worried they're just going to close the ticket as not being their issue. Anyone know of an engineer at CenturyLink/Lumen/Level3 or even Comcast that
Re: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around?
We have also seen the same behavior of intermittent customer complaint followed by issue resolving spontaneously. Our end customer has a tunnel to a supplier on Comcast with a return path via Lumen. The first ticket was opened on 9/23, has cleared and returned a few times. End customer worked with Cisco TAC and symptom is up to 30% ESP packet drop for periods from 5 minutes to 1 hour when the problem is active. From: NANOG on behalf of Mike Lewinski via NANOG Sent: Thursday, October 14, 2021 2:30 PM To: Brie ; nanog@nanog.org Subject: RE: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around? I can confirm this issue exists at several sites in the Denver area with this same IPSEC issue, all routing between Level3/Lumen and Comcast. I was told by one customer that it resolved late yesterday afternoon but I haven't been able to confirm that. Mike -Original Message- From: NANOG On Behalf Of Brie Sent: Thursday, October 14, 2021 10:43 AM To: nanog@nanog.org Subject: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around? Hi all, So, having a... frustrating issue going on. Long wall of text ahead as I explain. 1 x CenturyLink/Lumen fiber in Boise 1 x CenturyLink/Lumen fiber in Cheyenne 1 x Comcast biz fiber in Denver IPsec VPN tunnels between all three sites, w/ OSPF for routing failover (which unfortunately doesn't help in this situation). Two days ago, Cheyenne to Denver (.196) traffic (both tcp and udp) were an issue initially. Failed over to routing Cheyenne VPN through Boise while we opened ticket with CL. Yesterday, Boise to Denver (.196) traffic started having exact same issue. Tests from another CL fiber in Boise (my own circuit, with legacy IP space and BGP) to Denver (.196) did not show same issues. Path appeared clean. Traceroutes from Office Boise to Denver (.196) had a noticeable difference from Personal Boise to Denver (.196): Office Boise -> Denver (.196) -- 3: sea-edge-15.inet.qwest.net 4: lag-4.ear3.Seattle1.Level3.net 5: ae-2-52.ear2.seattle1.level3.net <-- This hop 6: be-203-pe01.seattle.wa.ibone.comcast.net Personal Boise -> Denver (.196) -- 4: sea-edge-15.inet.qwest.net 5: lag-25.ear2.Seattle1.Level3.net 6: be-203-pe01.seattle.wa.ibone.comcast.net On a whim, tracerouted to another Denver router IP address (.199, IP alias on same interface) from Boise Office, and traceroute matched the traceroute from Personal Boise to Denver (.196) traceroute. No packet loss. Swapped VPN tunnels over to using another ip on same router (.199), same interface physical and logical, in Denver, and VPN was working again normally. This morning though, Cheyenne to Denver (.199) is having problems, while Boise to Denver (.199) isn't (for now). Already spent most of last night working with my partner in Denver replacing nearly everything on the Denver side with no change. Tests from the router above the main Denver VPN endpoint (.196) do not show any kind of issues or packet loss to it, so it doesn't appear the problem is inside of our network. I'm not inclined to think this is a Comcast issue, but I can't be sure. This sounds almost like a load balancing hashing issue, with one link in the bond group having issues, somewhere in one of our upstream's networks. We'll be opening a ticket in a bit through normal channels with CenturyLink/Lumen, but we're worried they're just going to close the ticket as not being their issue. Anyone know of an engineer at CenturyLink/Lumen/Level3 or even Comcast that might want to take a stab at this? I can provide a lot more detail. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org/ http://www.ahbl.org
RE: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around?
I can confirm this issue exists at several sites in the Denver area with this same IPSEC issue, all routing between Level3/Lumen and Comcast. I was told by one customer that it resolved late yesterday afternoon but I haven't been able to confirm that. Mike -Original Message- From: NANOG On Behalf Of Brie Sent: Thursday, October 14, 2021 10:43 AM To: nanog@nanog.org Subject: Anyone from Level3/CenturyLink/Lumen, possibly Comcast around? Hi all, So, having a... frustrating issue going on. Long wall of text ahead as I explain. 1 x CenturyLink/Lumen fiber in Boise 1 x CenturyLink/Lumen fiber in Cheyenne 1 x Comcast biz fiber in Denver IPsec VPN tunnels between all three sites, w/ OSPF for routing failover (which unfortunately doesn't help in this situation). Two days ago, Cheyenne to Denver (.196) traffic (both tcp and udp) were an issue initially. Failed over to routing Cheyenne VPN through Boise while we opened ticket with CL. Yesterday, Boise to Denver (.196) traffic started having exact same issue. Tests from another CL fiber in Boise (my own circuit, with legacy IP space and BGP) to Denver (.196) did not show same issues. Path appeared clean. Traceroutes from Office Boise to Denver (.196) had a noticeable difference from Personal Boise to Denver (.196): Office Boise -> Denver (.196) -- 3: sea-edge-15.inet.qwest.net 4: lag-4.ear3.Seattle1.Level3.net 5: ae-2-52.ear2.seattle1.level3.net <-- This hop 6: be-203-pe01.seattle.wa.ibone.comcast.net Personal Boise -> Denver (.196) -- 4: sea-edge-15.inet.qwest.net 5: lag-25.ear2.Seattle1.Level3.net 6: be-203-pe01.seattle.wa.ibone.comcast.net On a whim, tracerouted to another Denver router IP address (.199, IP alias on same interface) from Boise Office, and traceroute matched the traceroute from Personal Boise to Denver (.196) traceroute. No packet loss. Swapped VPN tunnels over to using another ip on same router (.199), same interface physical and logical, in Denver, and VPN was working again normally. This morning though, Cheyenne to Denver (.199) is having problems, while Boise to Denver (.199) isn't (for now). Already spent most of last night working with my partner in Denver replacing nearly everything on the Denver side with no change. Tests from the router above the main Denver VPN endpoint (.196) do not show any kind of issues or packet loss to it, so it doesn't appear the problem is inside of our network. I'm not inclined to think this is a Comcast issue, but I can't be sure. This sounds almost like a load balancing hashing issue, with one link in the bond group having issues, somewhere in one of our upstream's networks. We'll be opening a ticket in a bit through normal channels with CenturyLink/Lumen, but we're worried they're just going to close the ticket as not being their issue. Anyone know of an engineer at CenturyLink/Lumen/Level3 or even Comcast that might want to take a stab at this? I can provide a lot more detail. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org/ http://www.ahbl.org
Anyone from Level3/CenturyLink/Lumen, possibly Comcast around?
Hi all, So, having a... frustrating issue going on. Long wall of text ahead as I explain. 1 x CenturyLink/Lumen fiber in Boise 1 x CenturyLink/Lumen fiber in Cheyenne 1 x Comcast biz fiber in Denver IPsec VPN tunnels between all three sites, w/ OSPF for routing failover (which unfortunately doesn't help in this situation). Two days ago, Cheyenne to Denver (.196) traffic (both tcp and udp) were an issue initially. Failed over to routing Cheyenne VPN through Boise while we opened ticket with CL. Yesterday, Boise to Denver (.196) traffic started having exact same issue. Tests from another CL fiber in Boise (my own circuit, with legacy IP space and BGP) to Denver (.196) did not show same issues. Path appeared clean. Traceroutes from Office Boise to Denver (.196) had a noticeable difference from Personal Boise to Denver (.196): Office Boise -> Denver (.196) -- 3: sea-edge-15.inet.qwest.net 4: lag-4.ear3.Seattle1.Level3.net 5: ae-2-52.ear2.seattle1.level3.net <-- This hop 6: be-203-pe01.seattle.wa.ibone.comcast.net Personal Boise -> Denver (.196) -- 4: sea-edge-15.inet.qwest.net 5: lag-25.ear2.Seattle1.Level3.net 6: be-203-pe01.seattle.wa.ibone.comcast.net On a whim, tracerouted to another Denver router IP address (.199, IP alias on same interface) from Boise Office, and traceroute matched the traceroute from Personal Boise to Denver (.196) traceroute. No packet loss. Swapped VPN tunnels over to using another ip on same router (.199), same interface physical and logical, in Denver, and VPN was working again normally. This morning though, Cheyenne to Denver (.199) is having problems, while Boise to Denver (.199) isn't (for now). Already spent most of last night working with my partner in Denver replacing nearly everything on the Denver side with no change. Tests from the router above the main Denver VPN endpoint (.196) do not show any kind of issues or packet loss to it, so it doesn't appear the problem is inside of our network. I'm not inclined to think this is a Comcast issue, but I can't be sure. This sounds almost like a load balancing hashing issue, with one link in the bond group having issues, somewhere in one of our upstream's networks. We'll be opening a ticket in a bit through normal channels with CenturyLink/Lumen, but we're worried they're just going to close the ticket as not being their issue. Anyone know of an engineer at CenturyLink/Lumen/Level3 or even Comcast that might want to take a stab at this? I can provide a lot more detail. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org/ http://www.ahbl.org
Level3/CenturyLink/Lumen in Denver, CO
Hello, If anybody has a circuit from Level3 (aka CenturyLink or, I guess, Lumen, now) in Denver, CO — particularly if you’re in the Iron Mountain facility on Brighton Blvd — and you are willing to briefly give me an iperf3 target, please contact me off-list. I’m trying to triangulate some throughput issues. Thanks, Matt
Re: CenturyLink -> Lumen
Matt Harris|Infrastructure Lead Engineer 816-256-5446|Direct Looking for something? Helpdesk Portal|Email Support|Billing Portal We build and deliver end-to-end IT solutions. On Wed, Sep 16, 2020 at 9:31 AM Tom Hill wrote: > On 16/09/2020 11:18, Matt Hoppes wrote: > > Quantum Fiber? Sounds like a misbranding. I highly doubt they are using > > Quantum technology. > > > Very prescient for when it becomes commercially possible though, eh? :) > How will they know if they have enough slack to patch a cut? They won't know until they observe it...
Re: CenturyLink -> Lumen
On 16/09/2020 11:18, Matt Hoppes wrote: > Quantum Fiber? Sounds like a misbranding. I highly doubt they are using > Quantum technology. Very prescient for when it becomes commercially possible though, eh? :) -- Tom
Re: CenturyLink -> Lumen
I think this is just someone trying to pull the stock price out of the dumps by branding themselves a “tech company”. There are still things from the TWTelecom days they haven’t finished integrating into the control panel, this should be fun watching them try to change the name at the same time. From: NANOG on behalf of "R. Leigh Hennig" Reply-To: "R. Leigh Hennig" Date: Wednesday, September 16, 2020 at 12:51 AM To: "nanog@nanog.org" Subject: CenturyLink -> Lumen https://www.fiercetelecom.com/telecom/centurylink-rebrands-re-defines-enterprise-sector-as-lumen-technology Curious. Any thoughts on how this changes their business approach, if any? Obviously something like this has to be planned far in advance, but I can’t help but wonder what impact the recent outage and bad press might have had on their plans here, possibly accelerating them? Probably not. But it’s an interesting move regardless. . | R. Leigh Hennig, Principal Network Architect ..| Markley Group https://markleygroup.com Sent from ProtonMail Mobile
Re: CenturyLink -> Lumen
Quantum Fiber? Sounds like a misbranding. I highly doubt they are using Quantum technology. That’s how Lowe’s got in a lawsuit for selling 8” boards that were 7.6” long and similar. > On Sep 16, 2020, at 12:53 AM, R. Leigh Hennig wrote: > > > https://www.fiercetelecom.com/telecom/centurylink-rebrands-re-defines-enterprise-sector-as-lumen-technology > > Curious. Any thoughts on how this changes their business approach, if any? > Obviously something like this has to be planned far in advance, but I can’t > help but wonder what impact the recent outage and bad press might have had on > their plans here, possibly accelerating them? Probably not. But it’s an > interesting move regardless. > > > . | R. Leigh Hennig, Principal Network Architect > ..| Markley Group https://markleygroup.com > > > Sent from ProtonMail Mobile
CenturyLink -> Lumen
https://www.fiercetelecom.com/telecom/centurylink-rebrands-re-defines-enterprise-sector-as-lumen-technology Curious. Any thoughts on how this changes their business approach, if any? Obviously something like this has to be planned far in advance, but I can’t help but wonder what impact the recent outage and bad press might have had on their plans here, possibly accelerating them? Probably not. But it’s an interesting move regardless. . | R. Leigh Hennig, Principal Network Architect ..| Markley Group https://markleygroup.com Sent from ProtonMail Mobile
