Re: [CloudOps] [outages] Major loss/latency on Twelve99 around NY/SJO

2024-08-15 Thread Phil Lavin via NANOG
2090.0%67 166.4 167.1 166.0 169.1 0.6 > On 15 Aug 2024, at 11:09, Phil Lavin wrote: > > Fwiw, still ongoing > > I raised an AWS ticket and they’re looking into it. If there’s any friendly > AWS Net Engineers on list, your help removing Twelve99 from route wo

Re: [CloudOps] [outages] Major loss/latency on Twelve99 around NY/SJO

2024-08-15 Thread Phil Lavin via NANOG
There's a planned SJC maintenance happening now. Expected end time is 2am > PST. > > On Thu, Aug 15, 2024, 1:38 AM Phil Lavin via Outages <mailto:outa...@outages.org>> wrote: >> It’s a good week for the Internet. Third outages list post this week :( >> >> Seei

Re: Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Phil Lavin via NANOG
I see the AWS->me path has just changed to Cogent. me->AWS path still appears to be Twelve99. Loss and latency has now subsided. Thanks > On 26 Jan 2024, at 08:19, Phil Lavin wrote: > > Hi Folks, > > Cross posting to outages and nanog. > > Can anybody from Twel

Re: Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Phil Lavin via NANOG
aku Ytti wrote: > > On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG wrote: > > >> 88.99.88.67 to 216.147.3.209: >> Host Loss% Snt Last Avg >> Best Wrst StDev >> 1. 10.88.10.254

Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Phil Lavin via NANOG
Hi Folks, Cross posting to outages and nanog. Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS? Traces below: 88.99.88.67 to 216.147.3.209: My traceroute [v0.94] phil (10.88.10.10) -> 216.147.3.209

AWS use1 reachability 16:11-17:36 UTC 15 Dec

2023-12-18 Thread Phil Lavin via NANOG
Hi Folks, Struggling to get any insight on this so hoping somebody else saw something. We lost about 15% of our traffic into our AWS use1 BYOIP ranges on Friday 15th Dec between 16:11 and 17:36 UTC. Customers reported being totally unable to reach us during the window. Unfortunately I didn’t ge

Re: SMTP-friendly VPS provider where I can also get a BGP feed

2023-09-26 Thread Phil Lavin via NANOG
> On 26 Sep 2023, at 11:09, Daniel Corbe wrote: > > I'm currently a vultr customer, but they're refusing to unblock port 25 on my > account. I've tried explaining my use case but no matter who I talk to over > there they just keep pointing me to their spam policy. I run an MTA in Hetzner. On

Re: Route Leak? AS11845 / Vox Telecom Ltd

2023-06-21 Thread Phil Lavin via NANOG
This seems to have been resolved and stable for the past 30 mins. Phil > On 21 Jun 2023, at 14:13, Phil Lavin wrote: > > Hi Folks, > > Seeing traffic from AWS use1 (216.147.2.235) to UK (62.3.100.19) has been > attempting to transit via AS11845 for the last few hours.

Route Leak? AS11845 / Vox Telecom Ltd

2023-06-21 Thread Phil Lavin via NANOG
Hi Folks, Seeing traffic from AWS use1 (216.147.2.235) to UK (62.3.100.19) has been attempting to transit via AS11845 for the last few hours. Looks like a route leak from Vox->AWS at DE-CIX. traceroute to 62.3.100.19 (62.3.100.19), 30 hops max, 60 byte packets 1 ec2-3-236-61-157.compute-1.ama

Re: AS37468 (Angola Cables) - Route Leak?

2023-05-25 Thread Phil Lavin via NANOG
Thanks, Mark > On 25 May 2023, at 13:34, Mark Tinka wrote: > > I've reached out to some known folk at Angola Cables. Will let you know if I > hear back. > > Mark. > > On 5/25/23 14:01, Phil Lavin via NANOG wrote: >> Cross-posting from outages list: >

AS37468 (Angola Cables) - Route Leak?

2023-05-25 Thread Phil Lavin via NANOG
Cross-posting from outages list: Hey Folks, Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69). Traffic was transiting via AS37468 (Angola Cables) in Coresite IX. Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on

Re: (Free)RADIUS Front-End

2021-09-17 Thread Phil Lavin via NANOG
It’s a very large hammer for the small nut you have to crack, but Zentyal (https://zentyal.com/community/) is worth a look. It’s a complete Linux OS that aims to provide a compatible alternative to MS Active Directory. FreeRadius is a component and, from what I remember, the GUI was excellent.

Re: OOB management options @ 60 Hudson & 1 Summer

2021-04-16 Thread Phil Lavin via NANOG
> On 15 Apr 2021, at 23:29, wrote: > > > Ha! “Surprised”? Well, offering OOB for a reasonable price could be a > differentiator for the savvy colo providers, but bean counters say: “Huh? If > customer X wants OOB, they can pay ~$300/mo for a cross-connect”. ~$300/mo > might seem an exaggera

Re: OOB management options @ 60 Hudson & 1 Summer

2021-04-16 Thread Phil Lavin via NANOG
> On 15 Apr 2021, at 23:14, Matthew Crocker wrote: > > I’m in DR space @ 60 Hudson and the Markeley MMR @ 1 Summer > > I’m surprised OOB bandwidth isn’t a feature for colocation providers. In --dayJob we were a customer of 1 Summer. OOB was provided by Markley in the form of a couple of L3 c

Re: Telehouse London Fire Evacuation Notice

2020-08-21 Thread Phil Lavin
TH ops are saying there’s a fire (or at least an alarm) in THN2. Will update as we find out more. > On 21 Aug 2020, at 21:24, Phil Lavin wrote: > > Hi folks, > > Did anyone else just get an email notice from Telehouse re fire evacuation? > Any idea if it’s legitima

Telehouse London Fire Evacuation Notice

2020-08-21 Thread Phil Lavin
Hi folks, Did anyone else just get an email notice from Telehouse re fire evacuation? Any idea if it’s legitimate or some sort of testing following the LD8 debacle? Phil

RE: Contact at Ubiquiti Networks?

2020-05-27 Thread Phil Lavin
> Where did you find this? Found out the hard way after buying and installing 20 of them. When a single node in a VC reboots, it starts switching traffic some seconds before it does STP so any loops that were previously blocked now flood - usually overloading the other 3400s on the network. Ap

RE: Contact at Ubiquiti Networks?

2020-05-26 Thread Phil Lavin
> Even the big guys like Juniper fail at basic functionality. Our brand new > MX204 fails to select the correct source address when doing ARP requests and > apparently that is a known will not fix. Apparently EX2300/EX3400 doesn't support STP when using Virtual Chassis and QFX51XX don't support

RE: LTE modem where I can control the MTU

2020-05-01 Thread Phil Lavin
> We have VZ wireless in the data center as a backup to our core > infrastructure. We have an issue where if packets have a large MTU they seem > to die. Does anyone know of a good 4G modem where I can set the MTU on the > cellular connection? I suspect it's a bit more complex than just changin

RE: Telehouse North 2 Temperatures

2020-04-08 Thread Phil Lavin
> I am seeing increased temperatures in our cage in THN2 and I am curious if > anyone else has noticed this as well. If you have gear in THN2 could you let > me know if you have seen an increase in Temps over the past week or so? Now that you mention it, yes. This is suite 260: https://pasteboa

RE: Recommended DDoS mitigation appliance?

2020-02-04 Thread Phil Lavin
> This sounds like a different model to me. Kentik I think averages out around > $500 per 10G per month I was talking about Imperva

RE: Recommended DDoS mitigation appliance?

2020-02-04 Thread Phil Lavin
> So is Imperva similar to how Kentik operates? What was it priced liked? It is a nice model as you don't need additional hardware or virtual appliances on-prem, which cuts down on the CAPEX cost. Like everyone else, they price the scrubbing based on your clean traffic levels. Price I have is ci

Colo

2019-12-17 Thread Phil Lavin
I'm looking for someone of a sales persuasion who sells small volume Colo in Equinix LA1-LA4, SV1, SV5, SV10 and/or SG2. Can anyone who does this please contact me off list? Thank you :)

RE: VDSL

2019-10-15 Thread Phil Lavin
> I discovered that the Budapest cable company was using VDLS to provide > services up to 500 megs into the buildings where my flats are located. VDSL > is a pretty old standard. I recollect people talking about it back in 1998. > Is it being heavily deployed in Last Mile networks state side?

RE: lots of traffic starting at 3 a.m. central time

2019-10-15 Thread Phil Lavin
> Anyone else see lots of traffic coming down starting at 3 a.m. central time ? > all of my internet connections showed strangely larger load for a few early > morning hours. Someone, on another list, mentioned a 70% increase in traffic to Akamai which seems to correlate with a new Fortnite re

RE: Spectrum (Charter) Fragmented UDP

2019-10-03 Thread Phil Lavin
> At some point over night on 30th September (i.e. the night going into 1st > October), we saw a number of Spectrum (Charter) customers stop handling > fragmented UDP packets To bring this thread to a close, Charter kindly investigated and fixed the issue. It was caused by a change to their net

RE: Spectrum (Charter) Fragmented UDP

2019-10-02 Thread Phil Lavin
> While we can say this should just work, the reality is, it's not very > reliably true and I would not build product or business on the assumption > that it works well. Yup. Understood. We can't get away from sending multi-packet messages. We try our best to keep SIP messages as small as possi

Spectrum (Charter) Fragmented UDP

2019-10-02 Thread Phil Lavin
At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets. This has manifested itself in such that the phones of affected customers are no longer receiving UDP SIP INVITE packets which

RE: NetworkLayer

2019-09-23 Thread Phil Lavin
Before the mob descends, I'll take the liberty of pointing you at this: https://archive.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf If the loss does not extend past a given hop to the end of the trace, it's not loss - it's probably a transit router rate limiting yo

RE: Mx204 alternative

2019-09-02 Thread Phil Lavin
> Does anyone use Juniper 0% finance? We're looking to upgrade from 4 x MX80s > and they are a big jump. Last I heard, it was $250k minimum order value so you'll struggle if you're only buying 4 units

Re: BGP router question

2019-08-08 Thread Phil Lavin
On 8 Aug 2019, at 22:40, Art Stephens mailto:asteph...@ptera.com>> wrote: Hope this is not too off topic but can any one advise if a Dell S4048-ON can support full ebgp routes? Datasheet (https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell-EMC-Networking-S4048-ON-S

RE: Colo in Africa

2019-07-16 Thread Phil Lavin
> just use the South Africa AWS region They don't have a Region there at present - only an Edge location. I believe one is in the works for launch next year.

RE: BGP prefix filter list

2019-05-15 Thread Phil Lavin
> We're an eyeball network. We accept default routes from our transit providers > so in theory there should be no impact on reachability. > I'm pretty concerned about things that I don't know due to inefficient > routing, e.g. customers hitting a public anycast DNS server in the wrong > locatio

RE: BGP prefix filter list

2019-05-15 Thread Phil Lavin
> We recently filtered out >=/24 prefixes since we're impacted by 768k day. What kind of network are you running? Doing such prefix filtering on an eyeball network strikes me as insane - you'd be cutting off customers from huge swathes of the Internet (including small companies like us) that don

RE: SNMP via proxy

2019-04-10 Thread Phil Lavin
> Going forward I was thinking of setting up a few hosts whose job would be to > simply relay SNMP traffic. This way moving forward we could hard code several > IP's and bounce all traffic through one of these IP's. You can Source NAT your monitoring servers through a single IP/pool of IPs on a

RE: Incoming SSDP UDP 1900 filtering

2019-03-25 Thread Phil Lavin
On Mon, 25 Mar 2019, marcel.duregards--- via NANOG wrote: > As SSDP is used with PnP for local LAN service discovery, we are > thinking of: > > 1) educate our client (take a lot of time) > 2) filter incoming SSDP packets (UDP port 1900 at least) in our bgp > border Looking back at logs for VoIP

RE: AS701/Verizon

2019-03-14 Thread Phil Lavin
> We're seeing consistent +100ms latency increases to Verizon customers in > Pennsylvania, during peak business hours for the past couple of weeks. Verizon reached out shortly after my e-mail to say they had resolved the issue - latency has been within normal bounds since. Many thanks :)

RE: AS701/Verizon

2019-03-12 Thread Phil Lavin
> or something else helpful :) Here's traceroutes, for those interested. Times are UTC. The issue is present to Verizon customers in both Pittsburgh and BlueBell. I don't have any other PA Verizon customers to reference against, though all of our other Verizon customers outside of PA look fine.

AS701/Verizon

2019-03-12 Thread Phil Lavin
We're seeing consistent +100ms latency increases to Verizon customers in Pennsylvania, during peak business hours for the past couple of weeks. If someone is able to assist, could they please contact me off-list?

RE: MX204 applications, (was about BGP RR design)

2019-02-15 Thread Phil Lavin
> They are normal 1st gen trio boxes, same as MPC1, MPC2, MPC3 originals were. > You may be confused about the fact that their control plane is freescale, > instead of intel. Sorry, yes - you're right. Re-convergence times are, however, still awful. Though if you're not handling a lot of routes

RE: MX204 applications, (was about BGP RR design)

2019-02-15 Thread Phil Lavin
> Anyone know why MX204 has so few ports? It seems like it only has WAN side > used, leaving FAB side entirely unused, throwing away 50% of free capacity. The usable port configs are also quite tricky. Juniper have had to make a tool to validate the configurations (https://apps.juniper.net/home/

RE: DNS Hijacking? - FiOS Northeast

2019-01-09 Thread Phil Lavin
> We are seeing DNS requests for A and to 8.8.8.8 come back with erroneous > replies resolving to 146.112.61.106 when sent via FiOS circuits in the > northeast. Anyone else seeing issues with DNS on FiOS in Northeast? Issue > started around 12:25 AM ET this morning and seems to be affecting

RE: Service Provider NetFlow Collectors

2019-01-02 Thread Phil Lavin
> Doesn't Kentik cost like $2000 a month minimum? We recently got a quote from Kentik and I fell off my chair. The annual cost was slightly more than the total upfront purchase cost of the hardware they were collecting Flow from and was significantly more than the total cost each year of runnin

RE: overages for power usage

2018-09-21 Thread Phil Lavin
> What kind of typical overage costs have you seen when a customer/you use more > than you've committed to? Telehouse London is 0.75 (GBP) per KWH of overage. Obviously it will depend on datacentre/country. Telehouse increase this annually at 2% above inflation measured against the RPI (last in

RE: Reaching out to ARIN members about their RPKI INVALID prefixes

2018-09-19 Thread Phil Lavin
> What about an one-off outreach effort? Makes sense to me. As someone who (at least pretends to) care, I was very much unaware of RPKI before seeing discussion about it on NANOG and #ix. That said, having recently done this with ARIN... they've got a long way to go before it's a simple process

RE: Massive Price Increase for X-conns at Telehouse Chelsea, NYC

2018-09-17 Thread Phil Lavin
> $350/mo seems to be standard. Our DCs are at $250.Seems more like they > held onto out of date pricing for a long time then realized it. For what it's worth, Telehouse London is around 30 USD/month for an x-connect within the same building. Our US datacentre (not Telehouse) on the other ha

RE: AS205869, AS57166: Featured Hijacker of the Month, July, 2018

2018-07-25 Thread Phil Lavin
> But this is a rather entirely different case. In this case, it seems that > one very notable peering that -did- in fact exist, between AS205869 and > AS6939, was not reported at all on the bgp.he.net page linked to above. HE usually learn these hijacked routes from IX peering and route server

RE: AS205869, AS57166: Featured Hijacker of the Month, July, 2018

2018-07-24 Thread Phil Lavin
> Dead for me via: > HE > NTT > COX Likewise here, via a bunch of other transits. I saw them from HE this morning but they appear to have been withdrawn now.

RE: AS3266: BitCanal hijack factory, courtesy of many connectivity providers

2018-07-09 Thread Phil Lavin
> Adjustments have been made and we are no longer accepting these. Indeed - I no longer see these routes from you. Thanks :)

RE: AS3266: BitCanal hijack factory, courtesy of many connectivity providers

2018-07-09 Thread Phil Lavin
> The only routes i can see now for 3266/197426 is two /24 v4 and one /29 v6 > that jumps on over to portugal through 1299 > (telia) -> 174 (cogent) -> 29003 (refertelecom / iptelecom). 6939 (HE) are still advertising the routes to their customers. That suggests that 197426 is still active on at

Re: Comcast

2018-06-29 Thread Phil Lavin
There’s a fairly wide reaching outage in the US. I imagine Comcast are already aware. > On 29 Jun 2018, at 18:56, Daniel Corbe wrote: > > Can someone from Comcast contact me off list? > > Your customers can’t reach my network right now. >

RE: Tunable QSFP Optics

2018-06-19 Thread Phil Lavin
> Does anyone know if any Single Mode QSFPs exist on the market that use > wavelengths other than 1310nm (either self tunable or factory tuned)? > I am looking to put more than one 40gb link on a fiber pair similar to using > DWDM OADMs for 1g & 10g but can't seem to find any qsfp optics that don

Re: 3rd party QSFP-100G-LR4-S for Cisco

2018-06-06 Thread Phil Lavin
I concur. Never bought Cisco 100G from them but the quality of the Juniper optics and other ancillary fibre stuff is great. > On 5 Jun 2018, at 20:58, Mitcheltree, Harold B wrote: > > FS.COM > > > --Pete > > > From: NANOG on behalf of Ryugo Kikuchi > > Se

RE: BGP Hijack/Sickness with AS4637

2018-06-05 Thread Phil Lavin
What is the relationship of 103.97.52.2 (Colocation Australia - Japan) to you? Is this, for example, a peering over an IX? If so, did you learn the route from route servers or do you peer directly with them? Phil -Original Message- From: NANOG On Behalf Of Alain Hebert Sent: 31 May 20

Re: Juniper BGP Convergence Time

2018-05-21 Thread Phil Lavin
Ask if they will configure BFD for you. I’ve not found many transit providers that will, but it’s worth a shot and it will lower failure detection to circa 1 second. > On 16 May 2018, at 17:49, Adam Kajtar wrote: > > I could use static routes but I noticed since I moved to full routes I have