West Coast Peering / Packet Loss issues between AS7018 (AT&T) and AS6461

[Alex Buie]

Since roughly 6:30AM local time on Tuesday we have been seeing
significantly increased packet loss (20-40%) and latency increase of
~30-40ms from customers on the west coast hosted out of AS7018
(AT&T/sbcglobal) towards our addresses announced through Zayo (AS6461).

Has anyone else noticed or seen this? We're working through our channels
with the carrier but the latency profile is such that it seems likely it
may be affecting others as well and I wanted to ask the group. Any
incidents that anyone is aware of which may be related?

Thanks,


*Alex Buie*Senior Cloud Operations Engineer

450 Century Pkwy # 100 Allen, TX 75013

D: 469-884-0225 | www.cytracom.com

Looking for contact at Microsoft optical networking

[Scott Campbell]

We are looking for a contact within the optical group at Microsoft 
networking to discuss a fiber outage event in the Chicago area during 
third party physical maintenance that resulted in accidental information 
disclosure between our organizations on or around 22 March, 2024. Our 
intent is to better understand exactly what happened and what can be 
done to mitigate potential fallout in the event that this happens again.


Many thanks,
Scott Campbell
ESnet Security Tech Lead

Re: West Coast Peering / Packet Loss issues between AS7018 (AT&T) and AS6461

[howard stearn]

To those interested ,
This is an ongoing issue for my users .
AT&T and Zayo have not responded to my requests .

AS7018   12.122.114.5
AS6461   64.125.12.117

>From June 4 , 2024 ,
Packet loss continues and has been going on between AT&T and Zayo/ABOVENET
all day .

According to DNS packet loss appears to be on a lag in SJC7 . ( Evocative
San Jose Data Center ? )
According to all traceroutes the IPs on each side of the packet loss are
AT&T 12.122.114.5 and ZAYO 64.125.12.117 .

When the traceroute confirms the IPs above are in the path ; packet loss to
or from any endpoint is greater than 15% .

So far I tried reaching out to technical support with both providers via
public and standard channels .

I’m asking for help passing this information along to AT&T or Zayo hoping
the community can provide some assistance to deliver this message because
their technical support can not .
Please forward this information along if you really do know the right
person to ask don’t just spam them with messages .
Don’t assume someone else will look at this and forward it to the correct
person .

Re: Open source Netflow analysis for monitoring AS-to-AS traffic

[Javier Gutierrez]

Hi everyone,
I've been trying to get Akvorado to work on my environmnet but I keep getting 
the flows to stop collecting, it seems like the issue is related to the number 
of exporters I have sending data, can someone please share the max number they 
have gotten to work and the flows/s rate without the system crashing?

Thanks in advance for your answers.

From: NANOG  on behalf of 
Steven Bakker 
Sent: Sunday, March 31, 2024 4:53 AM
To: Peter Phaal 
Cc: nanog@nanog.org 
Subject: Re: Open source Netflow analysis for monitoring AS-to-AS traffic


CAUTION: This email is from an external source. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.

Hi Peter,

Thanks for that link. I did read the spec, and while the definition itself is 
clear, the escape clause gives a lot of wiggle room:

"Hardware limitations may prevent an exact reporting of the underlying frame 
length, but an agent should attempt to be as accurate as possible."

I read that as, "the vendor will do whatever it pleases, and you should be 
grateful to receive a non-negative integer at all." I could be too cynical, 
though.

Anyway, this particular vendor does other funny things (such as sometimes 
stripping the q-tag headers from the sampled frame; throttling the frame 
sampling on the box, but not adjusting the sampling interval in the sFlow 
exports) that make it a true joy to work with this gear. ;-)

Cheers,

-- Steven

Re: West Coast Peering / Packet Loss issues between AS7018 (AT&T) and AS6461

[Mike Lyon]

Yep, im seeing it as well. I just opened a ticket with Zayo about an hour ago on it.-MikeOn Jun 5, 2024, at 14:23, Alex Buie  wrote:Since roughly 6:30AM local time on Tuesday we have been seeing significantly increased packet loss (20-40%) and latency increase of ~30-40ms from customers on the west coast hosted out of AS7018 (AT&T/sbcglobal) towards our addresses announced through Zayo (AS6461).Has anyone else noticed or seen this? We're working through our channels with the carrier but the latency profile is such that it seems likely it may be affecting others as well and I wanted to ask the group. Any incidents that anyone is aware of which may be related?Thanks,Alex BuieSenior Cloud Operations Engineer450 Century Pkwy # 100 Allen, TX 75013D: 469-884-0225 | www.cytracom.com

Any Zayo clue on the list?

[Mike Lyon]

Have a zayo packet loss issue and your NOC doesn't see anything wrong...

Host Loss% Snt Last Avg Best Wrst StDev 1.
ae15.cs4.sjc4.us.zip.zayo.com 94.3% 142 28.7 28.9 28.0 30.6 0.8 2.
ae8.cs2.sea1.us.zip.zayo.com 27.5% 142 30.8 30.5 27.5 41.6 3.9

Any help would be appreciated. Please ping me off list.

Thank You,
Mike

-- 
Mike Lyon
mike.l...@gmail.com
http://www.linkedin.com/in/mlyon

Re: Any Zayo clue on the list?

[Ryan Hamel]

Mike,

Do you have the full MTR? Packet loss in the middle of the trace means nothing 
if it is not persistent all the way to the end.

Ryan Hamel


From: NANOG  on behalf of Mike Lyon 

Sent: Wednesday, June 5, 2024 4:56 PM
To: NANOG list 
Subject: Any Zayo clue on the list?

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.


Have a zayo packet loss issue and your NOC doesn't see anything wrong...

Host Loss% Snt Last Avg Best Wrst StDev 1.
ae15.cs4.sjc4.us.zip.zayo.com 94.3% 142 28.7 28.9 28.0 30.6 0.8 2.
ae8.cs2.sea1.us.zip.zayo.com 27.5% 142 30.8 30.5 27.5 41.6 3.9

Any help would be appreciated. Please ping me off list.

Thank You,
Mike

--
Mike Lyon
mike.l...@gmail.com
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmlyon&data=05%7C02%7Cryan%40rkhtech.org%7Ca6a10508e6da4f64cbcd08dc85bb689b%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638532287160068907%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=4%2FczHZgV7KE6hVea2nFEThVBts%2BRJeEQd8UqzthloEA%3D&reserved=0