Re: Twelve99 / AWS usw2 significant loss

[Phil Lavin via NANOG]

I see the AWS->me path has just changed to Cogent. me->AWS path still appears 
to be Twelve99. Loss and latency has now subsided.

Thanks


> On 26 Jan 2024, at 08:19, Phil Lavin  wrote:
> 
> Hi Folks,
> 
> Cross posting to outages and nanog.
> 
> Can anybody from Twelve99 / AWS investigate high loss and latency around SJO 
> between Twelve99 and AWS?
> 
> Traces below:
> 
> 88.99.88.67 to 216.147.3.209:
> 
>My traceroute  [v0.94]
> phil (10.88.10.10) -> 216.147.3.209   
> 2024-01-26T08:16:16+
> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>  Packets   
> Pings
> Host   Loss%   Snt   Last   Avg  
> Best  Wrst StDev
> 1. 10.88.10.254 0.0%   1760.2   0.1   
> 0.1   0.3   0.1
> 2. fe-fw-01.lavin.me.uk 0.0%   1760.4   0.4   
> 0.2   0.7   0.1
> 3. 10.69.69.1   0.0%   1760.5   0.4   
> 0.2   0.7   0.1
> 4. static.225.8.4.46.clients.your-server.de 0.0%   1760.8   1.0   
> 0.5   8.2   1.0
> 5. core22.fsn1.hetzner.com  0.6%   1760.8   2.0   
> 0.7  39.0   4.6
> 6. static.213-239-254-234.clients.your-server.de0.0%   1763.2   3.8   
> 3.1  36.5   3.3
> 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5   
> 3.1  24.1   1.6
> 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
> 18.7  19.2   0.1
> 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
> 30.4  30.8   0.1
> 10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5  
> 99.3 100.1   0.2
> 11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
> 289.7 315.0   5.5
> 12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
> 500.7 539.7   8.4
> 13. (waiting for reply)
> …...
> 26. 216.147.3.209   59.1%   150  534.2 534.4 
> 513.5 549.7   8.1
> 
> 44.236.47.236 to 178.63.26.145:
> 
> jcasc-rtc01 (10.96.54.240) -> hc-ping.com   
> 2024-01-26T08:10:24+
> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>Packets   Pings
> Host Loss%   Snt   Last   Avg  
> Best  Wrst StDev
> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
> 0.2   0.4   0.0
> 2. ec2-34-221-151-225.us-west-2.compute.amazonaw  0.0%   2676.8  15.9   
> 0.7 114.1  24.1
> 3. (waiting for reply)
> 4. (waiting for reply)
> 5. (waiting for reply)
> 6. 241.0.2.5  0.0%   2670.4   0.4   
> 0.3   1.2   0.1
> 7. 242.0.42.195   0.0%   2671.2   1.7   
> 0.5  23.9   3.0
> 8. 240.2.144.12   0.0%   2675.2   5.2   
> 5.2  27.4   1.4
> 9. amazon-ic-375418.ip.twelve99-cust.net  0.0%   2675.3   5.2   
> 5.2   6.1   0.1
> 10. port-b4-link.ip.twelve99.net   0.0%   2675.3   5.5   
> 4.9  35.5   3.0
> 11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
> 5.6  11.8   0.5
> 12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
> 21.0  58.4   3.1
> 13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
> 21.3  86.2   6.5
> 14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
> 407.2 438.5   6.5
> 15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
> 481.6 491.1   3.9
> 16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
> 489.8 510.1   5.9
> 17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
> 504.4 561.7   7.6
> 18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
> 506.0 545.5   6.9
> 19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
> 498.5 531.7   6.6
> 20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
> 364.9 396.5   4.1
> 21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
> 506.9 531.3   6.1
> 
> 
> Thanks

Re: Twelve99 / AWS usw2 significant loss

[Phil Lavin via NANOG]

Thanks, ytti. I have raised a case with AWS but I expect it to be as 
unproductive as usual. In this type of issue, there is often a friendly 
Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. 
Hoping that to be the case this time, also.


> On 26 Jan 2024, at 08:40, Saku Ytti  wrote:
> 
> On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG  wrote:
> 
> 
>> 88.99.88.67 to 216.147.3.209:
>> Host   Loss%   Snt   Last   Avg  
>> Best  Wrst StDev
>> 1. 10.88.10.254 0.0%   1760.2   0.1  
>>  0.1   0.3   0.1
>> 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5  
>>  3.1  24.1   1.6
>> 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
>> 18.7  19.2   0.1
>> 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
>> 30.4  30.8   0.1
>> 10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5 
>>  99.3 100.1   0.2
>> 11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
>> 289.7 315.0   5.5
>> 12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
>> 500.7 539.7   8.4
> 
> This implies the problem is not on this path, because #10 is not
> experiencing it, possibly because it happens to return a packet via
> another option, but certainly shows the problem didn't happen in this
> direction yet at #10, but because #8 and #9 saw it, they already saw
> it on the other direction.
> 
> 
>> 44.236.47.236 to 178.63.26.145:
>> Host Loss%   Snt   Last   Avg  
>> Best  Wrst StDev
>> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
>> 0.2   0.4   0.0
>> 11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
>> 5.6  11.8   0.5
>> 12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
>> 21.0  58.4   3.1
>> 13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
>> 21.3  86.2   6.5
>> 14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
>> 407.2 438.5   6.5
>> 15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
>> 481.6 491.1   3.9
>> 16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
>> 489.8 510.1   5.9
>> 17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
>> 504.4 561.7   7.6
>> 18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
>> 506.0 545.5   6.9
>> 19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
>> 498.5 531.7   6.6
>> 20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
>> 364.9 396.5   4.1
>> 21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
>> 506.9 531.3   6.1
> 
> This suggests the congestion point is from sjo to nyk, in 1299, not AWS at 
> all.
> 
> You could try to fix SPORT/DPORT, and do several SPORT options, to see
> if loss goes away with some, to determine if all LAG members are full
> or just one.
> 
> 
> At any rate, this seems business as usual, sometimes internet is very
> lossy, you should contact your service provider, which I guess is AWS
> here, so they can contact their service provider or 1299.
> 
> -- 
>  ++ytti

Twelve99 / AWS usw2 significant loss

[Phil Lavin via NANOG]

Hi Folks,

Cross posting to outages and nanog.

Can anybody from Twelve99 / AWS investigate high loss and latency around SJO 
between Twelve99 and AWS?

Traces below:

88.99.88.67 to 216.147.3.209:

My traceroute  [v0.94]
phil (10.88.10.10) -> 216.147.3.209   
2024-01-26T08:16:16+
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
  Packets   
Pings
 Host   Loss%   Snt   Last   Avg  
Best  Wrst StDev
 1. 10.88.10.254 0.0%   1760.2   0.1   
0.1   0.3   0.1
 2. fe-fw-01.lavin.me.uk 0.0%   1760.4   0.4   
0.2   0.7   0.1
 3. 10.69.69.1   0.0%   1760.5   0.4   
0.2   0.7   0.1
 4. static.225.8.4.46.clients.your-server.de 0.0%   1760.8   1.0   
0.5   8.2   1.0
 5. core22.fsn1.hetzner.com  0.6%   1760.8   2.0   
0.7  39.0   4.6
 6. static.213-239-254-234.clients.your-server.de0.0%   1763.2   3.8   
3.1  36.5   3.3
 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5   
3.1  24.1   1.6
 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
18.7  19.2   0.1
 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
30.4  30.8   0.1
10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5  
99.3 100.1   0.2
11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
289.7 315.0   5.5
12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
500.7 539.7   8.4
13. (waiting for reply)
…...
26. 216.147.3.209   59.1%   150  534.2 534.4 
513.5 549.7   8.1

44.236.47.236 to 178.63.26.145:

jcasc-rtc01 (10.96.54.240) -> hc-ping.com   
2024-01-26T08:10:24+
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  
Best  Wrst StDev
 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
0.2   0.4   0.0
 2. ec2-34-221-151-225.us-west-2.compute.amazonaw  0.0%   2676.8  15.9   
0.7 114.1  24.1
 3. (waiting for reply)
 4. (waiting for reply)
 5. (waiting for reply)
 6. 241.0.2.5  0.0%   2670.4   0.4   
0.3   1.2   0.1
 7. 242.0.42.195   0.0%   2671.2   1.7   
0.5  23.9   3.0
 8. 240.2.144.12   0.0%   2675.2   5.2   
5.2  27.4   1.4
 9. amazon-ic-375418.ip.twelve99-cust.net  0.0%   2675.3   5.2   
5.2   6.1   0.1
10. port-b4-link.ip.twelve99.net   0.0%   2675.3   5.5   
4.9  35.5   3.0
11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
5.6  11.8   0.5
12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
21.0  58.4   3.1
13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
21.3  86.2   6.5
14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
407.2 438.5   6.5
15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
481.6 491.1   3.9
16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
489.8 510.1   5.9
17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
504.4 561.7   7.6
18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
506.0 545.5   6.9
19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
498.5 531.7   6.6
20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
364.9 396.5   4.1
21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
506.9 531.3   6.1


Thanks

AWS use1 reachability 16:11-17:36 UTC 15 Dec

[Phil Lavin via NANOG]

Hi Folks,

Struggling to get any insight on this so hoping somebody else saw something. We 
lost about 15% of our traffic into our AWS use1 BYOIP ranges on Friday 15th Dec 
between 16:11 and 17:36 UTC. Customers reported being totally unable to reach 
us during the window. Unfortunately I didn’t get a trace to an affected 
customer nor did any of our monitoring tools pick this up.

Did anybody see anything at the time and, if so, do you have any insights into 
what happened?


Thanks

Re: SMTP-friendly VPS provider where I can also get a BGP feed

[Phil Lavin via NANOG]

> On 26 Sep 2023, at 11:09, Daniel Corbe  wrote:
> 
> I'm currently a vultr customer, but they're refusing to unblock port 25 on my 
> account.  I've tried explaining my use case but no matter who I talk to over 
> there they just keep pointing me to their spam policy.


I run an MTA in Hetzner. Once you’ve paid a bill, you can raise a request to 
unblock port 25

Re: Route Leak? AS11845 / Vox Telecom Ltd

[Phil Lavin via NANOG]

This seems to have been resolved and stable for the past 30 mins.

Phil


> On 21 Jun 2023, at 14:13, Phil Lavin  wrote:
> 
> Hi Folks,
> 
> Seeing traffic from AWS use1 (216.147.2.235) to UK (62.3.100.19) has been 
> attempting to transit via AS11845 for the last few hours. Looks like a route 
> leak from Vox->AWS at DE-CIX.
> 
> traceroute to 62.3.100.19 (62.3.100.19), 30 hops max, 60 byte packets
> 1  ec2-3-236-61-157.compute-1.amazonaws.com (3.236.61.157)  2.505 ms  2.001 
> ms  1.988 ms
> 2  240.0.228.64 (240.0.228.64)  0.348 ms  0.443 ms  0.440 ms
> 3  240.0.228.89 (240.0.228.89)  0.334 ms  0.331 ms  0.430 ms
> 4  240.192.54.146 (240.192.54.146)  0.260 ms  0.310 ms  0.307 ms
> 5  240.0.228.57 (240.0.228.57)  0.250 ms  0.309 ms  0.305 ms
> 6  240.0.228.34 (240.0.228.34)  0.291 ms  0.298 ms  0.289 ms
> 7  240.0.48.14 (240.0.48.14)  0.832 ms  0.829 ms  0.826 ms
> 8  240.0.48.19 (240.0.48.19)  0.307 ms  0.400 ms  0.392 ms
> 9  240.192.15.51 (240.192.15.51)  0.343 ms  0.419 ms  0.456 ms
> 10  240.0.48.19 (240.0.48.19)  0.410 ms  0.510 ms  0.376 ms
> 11  240.0.48.4 (240.0.48.4)  0.369 ms  0.361 ms  0.325 ms
> 12  242.0.186.49 (242.0.186.49)  0.371 ms  0.395 ms  0.427 ms
> 13  52.93.28.113 (52.93.28.113)  3.347 ms  3.339 ms  3.336 ms
> 14  100.100.32.2 (100.100.32.2)  28.451 ms  28.448 ms  28.445 ms
> 15  100.91.192.41 (100.91.192.41)  6.698 ms  6.696 ms  6.693 ms
> 16  52.93.1.5 (52.93.1.5)  8.327 ms  8.286 ms  8.256 ms
> 17  52.93.51.36 (52.93.51.36)  6.958 ms  6.920 ms  6.947 ms
> 18  voxtelecom.co.za (206.82.105.31)  229.508 ms  229.811 ms  229.798 ms
> 19  41-193-120-5.vox.co.za (41.193.120.5)  228.144 ms  228.141 ms *
> 20  * * *
> 21  linx-2.zen.net.uk (195.66.236.158)  228.036 ms  228.154 ms *
> 22  * lag-6.p2.thn-lon.zen.net.uk (51.148.73.166)  223.587 ms *
> 23  * * *
> 24  * * *
> 25  * * *
> 26  * * *
> 27  * 62-3-100-19.dsl.in-addr.zen.co.uk (62.3.100.19)  230.499 ms  230.708 ms
> 
> Can anybody see the extent of the leak in your route tables? As it stands, my 
> home Internet is unbearable.
> 
> 
> Phil

Route Leak? AS11845 / Vox Telecom Ltd

[Phil Lavin via NANOG]

Hi Folks,

Seeing traffic from AWS use1 (216.147.2.235) to UK (62.3.100.19) has been 
attempting to transit via AS11845 for the last few hours. Looks like a route 
leak from Vox->AWS at DE-CIX.

traceroute to 62.3.100.19 (62.3.100.19), 30 hops max, 60 byte packets
 1  ec2-3-236-61-157.compute-1.amazonaws.com (3.236.61.157)  2.505 ms  2.001 ms 
 1.988 ms
 2  240.0.228.64 (240.0.228.64)  0.348 ms  0.443 ms  0.440 ms
 3  240.0.228.89 (240.0.228.89)  0.334 ms  0.331 ms  0.430 ms
 4  240.192.54.146 (240.192.54.146)  0.260 ms  0.310 ms  0.307 ms
 5  240.0.228.57 (240.0.228.57)  0.250 ms  0.309 ms  0.305 ms
 6  240.0.228.34 (240.0.228.34)  0.291 ms  0.298 ms  0.289 ms
 7  240.0.48.14 (240.0.48.14)  0.832 ms  0.829 ms  0.826 ms
 8  240.0.48.19 (240.0.48.19)  0.307 ms  0.400 ms  0.392 ms
 9  240.192.15.51 (240.192.15.51)  0.343 ms  0.419 ms  0.456 ms
10  240.0.48.19 (240.0.48.19)  0.410 ms  0.510 ms  0.376 ms
11  240.0.48.4 (240.0.48.4)  0.369 ms  0.361 ms  0.325 ms
12  242.0.186.49 (242.0.186.49)  0.371 ms  0.395 ms  0.427 ms
13  52.93.28.113 (52.93.28.113)  3.347 ms  3.339 ms  3.336 ms
14  100.100.32.2 (100.100.32.2)  28.451 ms  28.448 ms  28.445 ms
15  100.91.192.41 (100.91.192.41)  6.698 ms  6.696 ms  6.693 ms
16  52.93.1.5 (52.93.1.5)  8.327 ms  8.286 ms  8.256 ms
17  52.93.51.36 (52.93.51.36)  6.958 ms  6.920 ms  6.947 ms
18  voxtelecom.co.za (206.82.105.31)  229.508 ms  229.811 ms  229.798 ms
19  41-193-120-5.vox.co.za (41.193.120.5)  228.144 ms  228.141 ms *
20  * * *
21  linx-2.zen.net.uk (195.66.236.158)  228.036 ms  228.154 ms *
22  * lag-6.p2.thn-lon.zen.net.uk (51.148.73.166)  223.587 ms *
23  * * *
24  * * *
25  * * *
26  * * *
27  * 62-3-100-19.dsl.in-addr.zen.co.uk (62.3.100.19)  230.499 ms  230.708 ms

Can anybody see the extent of the leak in your route tables? As it stands, my 
home Internet is unbearable.


Phil

Re: AS37468 (Angola Cables) - Route Leak?

[Phil Lavin via NANOG]

Thanks, Mark



> On 25 May 2023, at 13:34, Mark Tinka  wrote:
> 
> I've reached out to some known folk at Angola Cables. Will let you know if I 
> hear back.
> 
> Mark.
> 
> On 5/25/23 14:01, Phil Lavin via NANOG wrote:
>> Cross-posting from outages list:
>> 
>> Hey Folks,
>> 
>> Seeing massive packet loss on routes from AWS to Hetzner today. First was 
>> AWS USW2 -> Hetzner (e.g. 88.99.88.69).
>> 
>> Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
>> 
>> Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix 
>> IX.
>> 
>> I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
>> 
>>My traceroute  [v0.94]
>> jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/>  
>> 2023-05-25T11:50:13+
>> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>> Packets   Pings
>>  Host Loss%   Snt   Last   Avg  Best  Wrst StDev
>>  1. ip-10-36-17-199.ap-southeast-  0.0%180.1   0.6   0.1   7.7   1.8
>>  2. ec2-18-141-171-15.ap-southeas  0.0%184.3  17.5   0.9  95.7  24.0
>> ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com 
>> <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/>
>>  3. (waiting for reply)
>>  4. (waiting for reply)
>>  5. (waiting for reply)
>>  6. (waiting for reply)
>>  7. (waiting for reply)
>>  8. 100.65.10.161  0.0%170.6   1.4   0.3   9.6   2.4
>> 100.65.11.65
>>  9. 150.222.108.75 0.0%172.1   3.0   1.0  13.1   3.3
>> 52.93.10.74
>> 10. 150.222.108.82 0.0%172.2   2.4   1.3  10.2   2.1
>> 11. 52.93.11.115   0.0%175.8   2.4   1.2   9.7   2.5
>> 52.93.10.185
>> 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/>  68.8%   
>>  17  268.6 279.0 262.4 312.8  19.8
>> 13. 102.219.127.3 93.8%17  420.0 420.0 420.0 420.0   0.0
>> 14. (waiting for reply)
>> 15. 195.66.227.20962.5%17  467.9 456.9 440.3 488.2  18.5
>> 16. core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0%
>> 17  455.6 464.1 455.6 486.3  14.8
>> 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/>   75.0%
>> 17  459.5 467.5 445.9 497.3  21.8
>> 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/>   68.8%
>> 17  478.4 441.0 390.0 478.4  32.0
>> 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/>87.5% 
>>17  460.4 454.2 448.1 460.4   8.7
>> 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0%17  
>> 433.3 424.2 382.7 441.9  27.9
>> 21. (waiting for reply)
>> 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/>75.0%
>> 17  436.8 438.6 385.0 493.3  44.2
>> 
>> I’ve been out of the BGP game for a few years now. Anybody have visibility 
>> of the extent of the leaks?
>> 
>> 
>> Phil
>> 
> 

AS37468 (Angola Cables) - Route Leak?

[Phil Lavin via NANOG]

Cross-posting from outages list:

Hey Folks,

Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS 
USW2 -> Hetzner (e.g. 88.99.88.69).

Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.

Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.

I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.

   My traceroute  [v0.94]
jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk   
2023-05-25T11:50:13+
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ip-10-36-17-199.ap-southeast-  0.0%180.1   0.6   0.1   7.7   1.8
 2. ec2-18-141-171-15.ap-southeas  0.0%184.3  17.5   0.9  95.7  24.0
ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com 

 3. (waiting for reply)
 4. (waiting for reply)
 5. (waiting for reply)
 6. (waiting for reply)
 7. (waiting for reply)
 8. 100.65.10.161  0.0%170.6   1.4   0.3   9.6   2.4
100.65.11.65
 9. 150.222.108.75 0.0%172.1   3.0   1.0  13.1   3.3
52.93.10.74
10. 150.222.108.82 0.0%172.2   2.4   1.3  10.2   2.1
11. 52.93.11.115   0.0%175.8   2.4   1.2   9.7   2.5
52.93.10.185
12. 37468-sg1-ix.equinix.com   68.8%
17  268.6 279.0 262.4 312.8  19.8
13. 102.219.127.3 93.8%17  420.0 420.0 420.0 420.0   0.0
14. (waiting for reply)
15. 195.66.227.20962.5%17  467.9 456.9 440.3 488.2  18.5
16. core6.par.hetzner.com  75.0%17  
455.6 464.1 455.6 486.3  14.8
17. core11.nbg1.hetzner.com    75.0%17 
 459.5 467.5 445.9 497.3  21.8
18. core23.fsn1.hetzner.com    68.8%17 
 478.4 441.0 390.0 478.4  32.0
19. ex9k1.dc1.fsn1.hetzner.com 87.5%
17  460.4 454.2 448.1 460.4   8.7
20. vmh02.lavin.me.uk  75.0%17  
433.3 424.2 382.7 441.9  27.9
21. (waiting for reply)
22. http-lb-01.lavin.me.uk 75.0%17  
436.8 438.6 385.0 493.3  44.2

I’ve been out of the BGP game for a few years now. Anybody have visibility of 
the extent of the leaks?


Phil

Re: (Free)RADIUS Front-End

[Phil Lavin via NANOG]

It’s a very large hammer for the small nut you have to crack, but Zentyal 
(https://zentyal.com/community/) is worth a look. It’s a complete Linux OS that 
aims to provide a compatible alternative to MS Active Directory. FreeRadius is 
a component and, from what I remember, the GUI was excellent.

Phil

> On 17 Sep 2021, at 18:26, Neil Hanlon  wrote:
> 
> 
> and I need more coffee... PacketFenCe
> 
> *sigh*
> 
> https://www.packetfence.org/
> 
> 
>> On Fri, Sep 17, 2021, 13:22 Neil Hanlon  wrote:
>> it's a bit more than just freeradius, but PacketFense is no-bs GPL software 
>> to do this, among much more.
>> 
>> I think it'd definitely do what you're looking to do
>> 
>> --Neil 
>> 
>>> On Fri, Sep 17, 2021, 12:30 Mark Tinka  wrote:
>>> Hi all.
>>> 
>>> I haven't been in the space in yonks, but I'm having to look into it for an 
>>> acquisition.
>>> 
>>> What's the latest on front-end panels for RADIUS, specifically, FreeRADIUS?
>>> 
>>> I fumbled around with Daloradius some years back, but mainly to manage some 
>>> pfSense captive portals for guest wi-fi VLAN's at the office.
>>> 
>>> I found these chaps who make some basic comparisons between themselves and 
>>> what is out there:
>>> 
>>> https://www.cloudradius.com/is-there-a-freeradius-gui/
>>> 
>>> Grateful to get feedback, on- and off-list, about what folk are doing with 
>>> this tech. nowadays. Enterpri$e options are welcome too, but essentially, 
>>> we are just looking for an easy on-premise (no cloud, please; I consider 
>>> RADIUS critical network infrastructure) pretty GUI system that can make the 
>>> engineers, NOC, provisioning and billing teams happy; and especially, the 
>>> customers, of course.
>>> 
>>> I don't trust myself with Google to avoid snake oil in my search :-).
>>> 
>>> All help appreciated. Thanks.
>>> 
>>> Mark.

Re: OOB management options @ 60 Hudson & 1 Summer

[Phil Lavin via NANOG]

> On 15 Apr 2021, at 23:29,   wrote:
> 
>  
> Ha! “Surprised”? Well, offering OOB for a reasonable price could be a 
> differentiator for the savvy colo providers, but bean counters say: “Huh? If 
> customer X wants OOB, they can pay ~$300/mo for a cross-connect”. ~$300/mo 
> might seem an exaggeration, but not for some of us. Even ~$150/mo is 
> ridiculous.

$300/month would be a bargain at One Summer

For what it’s worth, I really like Markley. Everybody that works there is 
fantastically helpful and they don’t cut corners on their infrastructure. Only 
gripe is X-Connect pricing

Re: OOB management options @ 60 Hudson & 1 Summer

[Phil Lavin via NANOG]

> On 15 Apr 2021, at 23:14, Matthew Crocker  wrote:
>  
> I’m in DR space @ 60 Hudson and the Markeley MMR @ 1 Summer
>  
> I’m surprised OOB bandwidth isn’t a feature for colocation providers.

In --dayJob we were a customer of 1 Summer. OOB was provided by Markley in the 
form of a couple of L3 circuits on SMF with some static IPs. I don’t recall the 
exact commercial relationship as I wasn’t in the company at the time it was 
negotiated but it was effectively supplied free of charge with the rack space. 
Cross-connects at 1 Summer are so damn expensive it’s impractical to take a 
feed from anybody else. That said, if you’re in the MMR then you probably have 
a lot more flexibility.

In other DCs we tended to either sub-lease rack space from a colo provider, as 
they could provide a feed from their adjacent network racks for very little 
cost, or we found a friendly network in the same suite to either take a cheap 
feed from or swap bandwidth with.

LTE reception in the majority of DCs is awful so I was never willing to trust 
it to work when needed.

Re: Telehouse London Fire Evacuation Notice

[Phil Lavin]

TH ops are saying there’s a fire (or at least an alarm) in THN2. Will update as 
we find out more.


> On 21 Aug 2020, at 21:24, Phil Lavin  wrote:
> 
> Hi folks,
> 
> Did anyone else just get an email notice from Telehouse re fire evacuation? 
> Any idea if it’s legitimate or some sort of testing following the LD8 debacle?
> 
> 
> Phil

Telehouse London Fire Evacuation Notice

[Phil Lavin]

Hi folks,

Did anyone else just get an email notice from Telehouse re fire evacuation? Any 
idea if it’s legitimate or some sort of testing following the LD8 debacle?


Phil

RE: Contact at Ubiquiti Networks?

[Phil Lavin]

> Where did you find this? 

Found out the hard way after buying and installing 20 of them. When a single 
node in a VC reboots, it starts switching traffic some seconds before it does 
STP so any loops that were previously blocked now flood - usually overloading 
the other 3400s on the network. Apparently this is just how the Broadcom chip 
works and cannot be fixed. That, twinned with some other significant stability 
issues, caused them to all go back for a refund. They did say a PR would be 
raised for this though I don't believe it has been - I can't find an external 
one, anyway.

The QFX51XX issue is currently a won't fix. Worked fine in 17.x, broke in 18.x. 
No documentation or reasoning for the change, other than giving the distinct 
impression that we don't spend enough for them to care.

RE: Contact at Ubiquiti Networks?

[Phil Lavin]

> Even the big guys like Juniper fail at basic functionality. Our brand new 
> MX204 fails to select the correct source address when doing ARP requests and 
> apparently that is a known will not fix.

Apparently EX2300/EX3400 doesn't support STP when using Virtual Chassis and 
QFX51XX don't support firewall filters when using VXLAN. Both cannot/will not 
fix. How I long to be in the spend category that makes vendors care about 
fundamental issues.

RE: LTE modem where I can control the MTU

[Phil Lavin]

> We have VZ wireless in the data center as a backup to our core 
> infrastructure. We have an issue where if packets have a large MTU they seem 
> to die. Does anyone know of a good 4G modem where I can set the MTU on the 
> cellular connection?

I suspect it's a bit more complex than just changing LTE MTU. The time when MTU 
matters in this situation (larger packets getting lost) is when DF bit is set 
on the packet - that's the case for all TCP data packets and it often crops up 
during TLS negotiations. Is that what you're seeing?

Consider the following scenario:

Your Server---Switch---LTE Router---T'Internet---Another 
Network---Other Server

If both "Your Server" and "Other Server" are connected to their relevant local 
networks at 1500 MTU then they will negotiate a TCP MSS slightly below 1500 
bytes (probably 1460 bytes???) because they have no concept of the path MTU. If 
the MTU between LTE Router and the Internet is below 1500 then the LTE Router 
will drop larger packets because it's not allowed to fragment them.

The solutions are:

:: Have the LTE Router reduce the MSS of TCP negotiation packets as they flow 
through it. This is the approach normally taken by any cheap DSL router so I'd 
think your current LTE router should be able to do this also
:: Have the LTE Router strip the DF bit from packets and fragment them anyway. 
I don't have any particular experience/opinions either way on this one so I'll 
leave it to others to comment/berate 
:: Implement path MTU discovery so your devices are aware of the path MTU and 
so set their MSS accordingly

RE: Telehouse North 2 Temperatures

[Phil Lavin]

> I am seeing increased temperatures in our cage in THN2 and I am curious if 
> anyone else has noticed this as well. If you have gear in THN2 could you let 
> me know if you have seen an increase in Temps over the past week or so?

Now that you mention it, yes. This is suite 260: 
https://pasteboard.co/J2T7d7k.png.

Looks like about +2.5c starting around midday on Sunday. It was warm outside 
before then so it feels unlikely to be as a result of outside temperature and 
fresh air cooling.

That said, none of our kit has alerted on temperature so it doesn't seem to be 
negatively affecting anything.

RE: Recommended DDoS mitigation appliance?

[Phil Lavin]

> This sounds like a different model to me. Kentik I think averages out around 
> $500 per 10G per month

I was talking about Imperva

RE: Recommended DDoS mitigation appliance?

[Phil Lavin]

> So is Imperva similar to how Kentik operates? What was it priced liked?

It is a nice model as you don't need additional hardware or virtual appliances 
on-prem, which cuts down on the CAPEX cost. Like everyone else, they price the 
scrubbing based on your clean traffic levels. Price I have is circa $73,000 a 
year for 250mbit clean traffic and circa $94,000 a year for 500mbit clean 
traffic. Reasonably good value if you get attacked a lot - a very expensive 
insurance policy if not. Yearly pricing is broadly on par with Radware, Arbor 
and A10 (Verisign).

Colo

[Phil Lavin]

I'm looking for someone of a sales persuasion who sells small volume Colo in 
Equinix LA1-LA4, SV1, SV5, SV10 and/or SG2. Can anyone who does this please 
contact me off list?

Thank you :)

RE: VDSL

[Phil Lavin]

> I discovered that the Budapest cable company was using VDLS to provide 
> services up to 500 megs into the buildings where my flats are located. VDSL 
> is a pretty old standard. I recollect people talking about it back in 1998. 

> Is it being heavily deployed in Last Mile networks state side?

DSL on the whole seems pretty unpopular in the USA. VDSL itself is a fairly old 
standard but it's been enhanced over the years to provide bandwidths up to 
300mbit on a single twisted copper pair, albeit over relatively short distances.

DSL (these days, specifically VDSL2) is extremely popular and widely used 
within the UK because almost every home has a single twisted pair going into it 
for a POTS phone line. It made sense to run services over this than to re-cable 
25 million homes. A (very) slow FTTH rollout is under way but what seems to be 
getting more traction is a rollout of G.Fast which currently boasts speeds of 
up to 500mbit over short distances (< 100m), still on a single twisted copper 
pair. This may be what you're getting as VDSL2 won't push to 500mbit over any 
sensible distance.

I can only speculate on why they decided to use DSL in your building - if it 
has legacy POTS infrastructure to each apartment, it would make some sense. If 
not, who knows...

RE: lots of traffic starting at 3 a.m. central time

[Phil Lavin]

> Anyone else see lots of traffic coming down starting at 3 a.m. central time ? 
>  all of my internet connections showed strangely larger load for a few early 
> morning hours.

Someone, on another list, mentioned a 70% increase in traffic to Akamai which 
seems to correlate with a new Fortnite release

RE: Spectrum (Charter) Fragmented UDP

[Phil Lavin]

> At some point over night on 30th September (i.e. the night going into 1st 
> October), we saw a number of Spectrum (Charter) customers stop handling 
> fragmented UDP packets

To bring this thread to a close, Charter kindly investigated and fixed the 
issue. It was caused by a change to their network that occurred on 1st October. 
Thanks to those involved for their help.

RE: Spectrum (Charter) Fragmented UDP

[Phil Lavin]

> While we can say this should just work, the reality is, it's not very 
> reliably true and I would not build product or business on the assumption 
> that it works well.

Yup. Understood. We can't get away from sending multi-packet messages. We try 
our best to keep SIP messages as small as possible though sometimes certain 
optional features required by customers push it beyond their MTU. We're also 
starting to see decreasing MTUs as customers deploy various SD-WAN solutions 
and it's tough to keep up with these when you're already teetering on the edge 
of what used to be considered a fairly common minimum MTU value.

We can, of course, get away from using UDP. We can and do run SIP over TCP and 
indeed over TLS on TCP though the stateful nature of TCP often makes this 
undesirable. We see a lot of SIP phone implementations that do not handle TCP 
connection failures very well and result in a loss of calls for a period of 
minutes if this happens, as they take a while to notice the connection has 
dropped and should be re-established. Pros and cons of each.

If anyone has any specific information about Spectrum CPE changes or indeed any 
contacts who may be able to interrogate this internally within Spectrum that 
would be appreciated.

Spectrum (Charter) Fragmented UDP

[Phil Lavin]

At some point over night on 30th September (i.e. the night going into 1st 
October), we saw a number of Spectrum (Charter) customers stop handling 
fragmented UDP packets. This has manifested itself in such that the phones of 
affected customers are no longer receiving UDP SIP INVITE packets which exceed 
whatever their WAN side MTU is. We've so far had 6 customers report the issue - 
we can see that the last call on 30th September worked and the first and 
subsequent calls on 1st October failed.

Is anyone aware of an update to CPE devices pushed out that night which may 
have broken their ability to handle IP fragmentation?

RE: NetworkLayer

[Phil Lavin]

Before the mob descends, I'll take the liberty of pointing you at this: 
https://archive.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf

If the loss does not extend past a given hop to the end of the trace, it's not 
loss - it's probably a transit router rate limiting your trace packets.

RE: Mx204 alternative

[Phil Lavin]

> Does anyone use Juniper 0% finance? We're looking to upgrade from 4 x MX80s 
> and they are a big jump.

Last I heard, it was $250k minimum order value so you'll struggle if you're 
only buying 4 units

Re: BGP router question

[Phil Lavin]

On 8 Aug 2019, at 22:40, Art Stephens 
mailto:asteph...@ptera.com>> wrote:

Hope this is not too off topic but can any one advise if a Dell S4048-ON can 
support full ebgp routes?

Datasheet 
(https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell-EMC-Networking-S4048-ON-Spec-Sheet.pdf)
 says 128k v4 routes, so no

RE: Colo in Africa

[Phil Lavin]

> just use the South Africa AWS region

They don't have a Region there at present - only an Edge location. I believe 
one is in the works for launch next year.

RE: BGP prefix filter list

[Phil Lavin]

> We're an eyeball network. We accept default routes from our transit providers 
> so in theory there should be no impact on reachability.

> I'm pretty concerned about things that I don't know due to inefficient 
> routing, e.g. customers hitting a public anycast DNS server in the wrong 
> location resulting in Geolocation issues.

Ah! Understood. The default route(s) was the bit I missed. Makes a lot of sense 
if you can't justify buying new routers.

Have you seen issues with Anycast routing thus far? One would assume that 
routing would still be fairly efficient unless you're picking up transit from 
non-local providers over extended L2 links.

RE: BGP prefix filter list

[Phil Lavin]

> We recently filtered out >=/24 prefixes since we're impacted by 768k day.

What kind of network are you running? Doing such prefix filtering on an eyeball 
network strikes me as insane - you'd be cutting off customers from huge swathes 
of the Internet (including small companies like us) that don't have large IPv4 
sequential allocations.

RE: SNMP via proxy

[Phil Lavin]

> Going forward I was thinking of setting up a few hosts whose job would be to 
> simply relay SNMP traffic. This way moving forward we could hard code several 
> IP's and bounce all traffic through one of these IP's.

You can Source NAT your monitoring servers through a single IP/pool of IPs on a 
NAT enabled router. We do something similar with RADIUS where the RADIUS server 
requires a single source IP for each client but the clients don't have fixed 
IPs (containers in AWS)

RE: Incoming SSDP UDP 1900 filtering

[Phil Lavin]

On Mon, 25 Mar 2019, marcel.duregards--- via NANOG wrote:
> As SSDP is used with PnP for local LAN service discovery, we are 
> thinking of:
>
> 1) educate our client (take a lot of time)
> 2) filter incoming SSDP packets (UDP port 1900 at least) in our bgp 
> border

Looking back at logs for VoIP calls over the past week (we are a small B2B 
telco), I see a fair number of RTP streams originating from UDP port 1900 on 
the customer side. Such filtering at the provider edge would have caused one 
way audio on these calls

RE: AS701/Verizon

[Phil Lavin]

> We're seeing consistent +100ms latency increases to Verizon customers in 
> Pennsylvania, during peak business hours for the past couple of weeks.

Verizon reached out shortly after my e-mail to say they had resolved the issue 
- latency has been within normal bounds since. Many thanks :)

RE: AS701/Verizon

[Phil Lavin]

> or something else helpful :)

Here's traceroutes, for those interested. Times are UTC. The issue is present 
to Verizon customers in both Pittsburgh and BlueBell. I don't have any other PA 
Verizon customers to reference against, though all of our other Verizon 
customers outside of PA look fine.

phil@debian:~$ mtr -zwc1 108.16.123.123
Start: Tue Mar 12 00:19:43 2019
HOST: debianLoss%   Snt   Last   
Avg  Best  Wrst StDev
  1. AS32334  192.30.36.123   0.0% 12.7 
  2.7   2.7   2.7   0.0
  2. AS???10.11.11.1  0.0% 12.6 
  2.6   2.6   2.6   0.0
  3. AS2914   129.250.199.37  0.0% 11.5 
  1.5   1.5   1.5   0.0
  4. AS2914   ae-6.r24.nycmny01.us.bb.gin.ntt.net 0.0% 19.4 
  9.4   9.4   9.4   0.0
  5. AS2914   ae-1.r08.nycmny01.us.bb.gin.ntt.net 0.0% 16.6 
  6.6   6.6   6.6   0.0
  6. AS701et-7-0-5.BR3.NYC4.ALTER.NET 0.0% 18.5 
  8.5   8.5   8.5   0.0
  7. AS??????100.0 10.0 
  0.0   0.0   0.0   0.0
  8. AS701ae203-0.PHLAPA-VFTTP-302.verizon-gni.net0.0% 1  137.2 
137.2 137.2 137.2   0.0
  9. AS701static-108-16-123-123.phlapa.fios.verizon.net   0.0% 1  118.4 
118.4 118.4 118.4   0.0

phil@debian:~$ mtr -zwc1 108.16.123.123
Start: Tue Mar 12 07:48:25 2019
HOST: debianLoss%   Snt   Last   
Avg  Best  Wrst StDev
  1. AS32334  192.30.36.123   0.0% 12.7 
  2.7   2.7   2.7   0.0
  2. AS???10.11.11.1  0.0% 11.0 
  1.0   1.0   1.0   0.0
  3. AS2914   129.250.199.37  0.0% 12.9 
  2.9   2.9   2.9   0.0
  4. AS2914   ae-6.r24.nycmny01.us.bb.gin.ntt.net 0.0% 17.2 
  7.2   7.2   7.2   0.0
  5. AS2914   ae-1.r08.nycmny01.us.bb.gin.ntt.net 0.0% 19.1 
  9.1   9.1   9.1   0.0
  6. AS701et-7-0-5.BR3.NYC4.ALTER.NET 0.0% 17.1 
  7.1   7.1   7.1   0.0
  7. AS??????100.0 10.0 
  0.0   0.0   0.0   0.0
  8. AS701ae203-0.PHLAPA-VFTTP-302.verizon-gni.net0.0% 1   14.7 
 14.7  14.7  14.7   0.0
  9. AS701static-108-16-123-123.phlapa.fios.verizon.net   0.0% 1   17.8 
 17.8  17.8  17.8   0.0

Smokeping graph at https://ibb.co/g4VQR8k  

AS701/Verizon

[Phil Lavin]

We're seeing consistent +100ms latency increases to Verizon customers in 
Pennsylvania, during peak business hours for the past couple of weeks.

If someone is able to assist, could they please contact me off-list?

RE: MX204 applications, (was about BGP RR design)

[Phil Lavin]

> They are normal 1st gen trio boxes, same as MPC1, MPC2, MPC3 originals were. 
> You may be confused about the fact that their control plane is freescale, 
> instead of intel.

Sorry, yes - you're right. Re-convergence times are, however, still awful. 
Though if you're not handling a lot of routes, that may not be a huge problem 
for you

RE: MX204 applications, (was about BGP RR design)

[Phil Lavin]

> Anyone know why MX204 has so few ports? It seems like it only has WAN side 
> used, leaving FAB side entirely unused, throwing away 50% of free capacity.

The usable port configs are also quite tricky. Juniper have had to make a tool 
to validate the configurations (https://apps.juniper.net/home/port-checker/). 
For example, using 4x40G disables all of the 10G ports however using 3x40G and 
1x100G gives you all of the 10G ports

> MX80/MX104 have both sides for revenue ports.

They are, however, not Trio - rather just commodity CPUs. Routing 
re-convergence times are shockingly high - in the region of 5-10 minutes for 
MX80 with a full table vs 30 seconds (ish) for 204

> I would GLADLY take 50% more ports in MX204, without taking any more PPS or 
> QoS bandwidth.

You can add switches (EX or QFX) as line cards using Fusion, to add more port 
density. I've heard some good things about Fusion, though I'm always wary of 
proprietary clustering technology having been bitten by VC a few times. You can 
also just trunk some VLANs up to switches if you don't want to buy the Fusion 
license

RE: DNS Hijacking? - FiOS Northeast

[Phil Lavin]

> We are seeing DNS requests for A and  to 8.8.8.8 come back with erroneous 
> replies resolving to 146.112.61.106 when sent via FiOS circuits in the 
> northeast. Anyone else seeing issues with DNS on FiOS in Northeast? Issue 
> started around 12:25 AM ET this morning and seems to be affecting customers 
> in PA, RI, etc.. 

146.112.61.106 appears to be an Anycast IP served by OpenDNS when pages are 
blocked by the Cisco Umbrella service - 
https://support.opendns.com/hc/en-us/articles/227986927-What-are-the-Cisco-Umbrella-Block-Page-IP-Addresses-

Are you sure the queries are going to Google 8.8.8.8 and not OpenDNS?

What URL(s) are you seeing this on?

Do you have a traceroute to 8.8.8.8 from an affected site?

RE: Service Provider NetFlow Collectors

[Phil Lavin]

> Doesn't Kentik cost like $2000 a month minimum?

We recently got a quote from Kentik and I fell off my chair. The annual cost 
was slightly more than the total upfront purchase cost of the hardware they 
were collecting Flow from and was significantly more than the total cost each 
year of running the network (when factoring in power, cross-connects, transit, 
etc.). I'm told that collecting Flow and BGP from two routers which both do < 
10Gbit is "~$32K USD per year for Kentik Pro" - even that feels 
disproportionally expensive. It does look like an excellent product though I'm 
struggling to find a business case that justifies the cost on relatively small 
(sub 2 Tbit/s) networks.

RE: overages for power usage

[Phil Lavin]

> What kind of typical overage costs have you seen when a customer/you use more 
> than you've committed to?

Telehouse London is 0.75 (GBP) per KWH of overage. Obviously it will depend on 
datacentre/country. Telehouse increase this annually at 2% above inflation 
measured against the RPI (last increase was 5.27%)

RE: Reaching out to ARIN members about their RPKI INVALID prefixes

[Phil Lavin]

> What about an one-off outreach effort?

Makes sense to me. As someone who (at least pretends to) care, I was very much 
unaware of RPKI before seeing discussion about it on NANOG and #ix.

That said, having recently done this with ARIN... they've got a long way to go 
before it's a simple process (like RIPE). Submitting numerous tickets over a 3 
day period doesn't strike me as particularly efficient. If outreach was done 
and widely taken up, I'd think ARIN's help desk will struggle to meet the 
demand. If this is the case and it's a multi-week process to get RPKI set up, 
it would be expected that people will give up part way through the process.

RE: Massive Price Increase for X-conns at Telehouse Chelsea, NYC

[Phil Lavin]

> $350/mo seems to be standard. Our DCs are at $250.Seems more like they 
> held onto out of date pricing for a long time then realized it.

For what it's worth, Telehouse London is around 30 USD/month for an x-connect 
within the same building. Our US datacentre (not Telehouse) on the other hand 
is around 200 USD/month. It's always felt disproportionally expensive but maybe 
those kind of prices are expected for the North America region.

RE: AS205869, AS57166: Featured Hijacker of the Month, July, 2018

[Phil Lavin]

> But this is a rather entirely different case.  In this case, it seems that 
> one very notable peering that -did- in fact exist, between AS205869 and 
> AS6939, was not reported at all on the bgp.he.net page linked to above.

HE usually learn these hijacked routes from IX peering and route servers. 
bgp.he.net tends not to report on IX peering. To their credit, HE have an 
extremely open peering policy. To their detriment, this means their customers 
see the large majority of hijacked routes due to poor filtering on many IX 
route servers.

RE: AS205869, AS57166: Featured Hijacker of the Month, July, 2018

[Phil Lavin]

> Dead for me via:
> HE
> NTT
> COX

Likewise here, via a bunch of other transits. I saw them from HE this morning 
but they appear to have been withdrawn now.

RE: AS3266: BitCanal hijack factory, courtesy of many connectivity providers

[Phil Lavin]

> Adjustments have been made and we are no longer accepting these.

Indeed - I no longer see these routes from you. Thanks :)

RE: AS3266: BitCanal hijack factory, courtesy of many connectivity providers

[Phil Lavin]

> The only routes i can see now for 3266/197426 is two /24 v4 and one /29 v6 
> that jumps on over to portugal through 1299
> (telia) -> 174 (cogent) -> 29003 (refertelecom / iptelecom).

6939 (HE) are still advertising the routes to their customers. That suggests 
that 197426 is still active on at least one IX. 

Re: Comcast

[Phil Lavin]

There’s a fairly wide reaching outage in the US. I imagine Comcast are already 
aware.


> On 29 Jun 2018, at 18:56, Daniel Corbe  wrote:
> 
> Can someone from Comcast contact me off list?
> 
> Your customers can’t reach my network right now.
> 

RE: Tunable QSFP Optics

[Phil Lavin]

> Does anyone know if any Single Mode QSFPs exist on the market that use 
> wavelengths other than 1310nm (either self tunable or factory tuned)?
> I am looking to put more than one 40gb link on a fiber pair similar to using 
> DWDM OADMs for 1g & 10g but can't seem to find any qsfp optics that don't use 
> 1310nm.

I've probed our optics vendor about this in the past when trying to achieve 
something similar. Although they state 1310nm, they actually run at 4 different 
10G wavelengths (one of which is 1310) and as such are not suitable for 
Mux/Demuxing.

Re: 3rd party QSFP-100G-LR4-S for Cisco

[Phil Lavin]

I concur. Never bought Cisco 100G from them but the quality of the Juniper 
optics and other ancillary fibre stuff is great.


> On 5 Jun 2018, at 20:58, Mitcheltree, Harold B  wrote:
> 
> FS.COM
> 
> 
> --Pete
> 
> 
> From: NANOG  on behalf of Ryugo Kikuchi 
> 
> Sent: Tuesday, May 29, 2018 7:48:16 AM
> To: nanog@nanog.org
> Subject: 3rd party QSFP-100G-LR4-S for Cisco
> 
> Hey all,
> 
> Does anyone have a recommended model of 3rd party's "QSFP-100G-LR4-S" for
> Cisco ASR and Nexus?
> 
> Cisco's original 100G SFP costs us an arm and a leg, so we want to try to
> use 3rd party 100g SFP.
> But we are not sure which manufacturer's SFP is reliable or has good
> performance.
> 
> Does anyone have experience?
> 
> Many thanks,
> 
> Roy

RE: BGP Hijack/Sickness with AS4637

[Phil Lavin]

What is the relationship of 103.97.52.2 (Colocation Australia - Japan) to you? 
Is this, for example, a peering over an IX? If so, did you learn the route from 
route servers or do you peer directly with them?


Phil

-Original Message-
From: NANOG  On Behalf Of Alain Hebert
Sent: 31 May 2018 14:50
To: nanog@nanog.org
Subject: Re: BGP Hijack/Sickness with AS4637

     Hi,

     Well bad news on the ColoAU front, they refused to cooperate.

     We'll pushback thru our GTT accounts...  But I'm running out of ideas.

     If anyone has any good ideas how to proceed at this point feel free to 
share =D.

-
Alain Hebertaheb...@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 05/29/18 16:31, Chris Conn wrote:
> Hello,
>
> I am the contact for AS16532.
>
> We never announced nor are we currently advertising this prefix as we 
> are not a transit AS for anyone.  As well, it seems to appear and 
> disappear from AS63956 looking glass.  According to that LG, the route 
> changed 6d ago, and is *still currently visible* at this very moment;
>
> https://lg.coloau.com.au/
>
> Command: show route 18.29.238.0 protocol bgp table 
> vrf-international.inet.0 active-path
>
> vrf-international.inet.0: 696764 destinations, 2288960 routes (696480 
> active, 0 holddown, 103994 hidden)
> + = Active Route, - = Last Active, * = Both
>
> 18.29.238.0/23 *[BGP/170] 6d 01:06:11, localpref 90, from 103.97.52.2
>AS path: 4637 3257 29909 16532 16532 16532 
> 16532 I, validation-state: unverified
>
>
> AS16532 is not announcing this prefix.  We have a strict prefix-list that is 
> applied to all sessions.  As well, AS29909 is filtering us using our 
> announced AS-SETS/RPSL to avoid us the ability to do anything dumb.  And 
> lastly, our announcements are being filtered by AS3257 as we are required to 
> provide them via LOA.
>
> There is still something wrong somewhere that is injecting this path, anyone 
> have a LG pointed to AS4637 seeing this prefix announced with AS16532 in the 
> AS path?
>
> I doubt that AS29909 bouncing its BGP session with AS3257 (GTT) would 
> change anything, as I am not seeing this prefix in their route-server
>
> pub...@route-server.as3257.net-re0> show route 18.29.238.0 protocol 
> bgp active-path
>
> inet.0: 691667 destinations, 11752983 routes (691665 active, 1 
> holddown, 1 hidden)
> + = Active Route, - = Last Active, * = Both
>
> 18.29.0.0/16   *[BGP/170] 3w4d 11:42:33, MED 0, localpref 100, from 
> 213.200.87.23
>AS path: 3257 174 3 I, validation-state: unverified
>  > to 141.136.111.13 via xe-1/0/0.0
>
> {master}
> pub...@route-server.as3257.net-re0>
>
>
> {master}
> pub...@route-server.as3257.net-re0> show route 18.29.238.0 protocol 
> bgp | find 16532
>
> Pattern not found
> {master}
>
>
>
> So whatever is happening, its not at AS16532, AS29909 nor AS3257 that I can 
> find.
>
>
> Chris Conn
> AS16532
>
>
>
>
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tom Paseka 
> via NANOG
> Sent: Friday, May 25, 2018 6:01 PM
> To: Nikolas Geyer 
> Cc: NANOG list 
> Subject: Re: BGP Hijack/Sickness with AS4637
>
> This looks like a route that has been cached by some ISPs/routers even though 
> a withdrawal has actually happened.
>
> If you actually forward packets a long the path, you'll see its not following 
> the AS Path suggested, instead the real route that it should be.
> Bouncing your session with 4637 would likely clear this.
>
> -Tom
>
> On Fri, May 25, 2018 at 11:59 AM, Nikolas Geyer  wrote:
>
>> Greetings!
>>
>> Actually, what you have provided below shows the exact opposite. It 
>> shows ColoAU have received the route from 4637 who have received it 
>> from 3257 who have received it from 29909 who have received it from
>> 16532 who originated it. It infers nothing about who 16532 found the route 
>> to come from.
>>
>> It is evident that GTT are advertising that route to Telstra Global 
>> :)
>>
>> Regards,
>> Nik.
>>
>>>  And I'm pretty sure AS3257 (GTT ) is in the same boat as 
>>> us, as
>> they're not the one advertising those routes to AS4637
>>>  AS16532 found it to come from AS4637 as you can see from this 
>>> ColoAU
>> LG output below
>>>
>>> - https://lg.coloau.com.au/
>>>
>>> vrf-international.inet.0: 696533 destinations, 2248101 routes
>>> (696249
>> active, 0 holddown, 103835 hidden)
>>> + = Active Route, - = Last Active, * = Both
>>>
>>> 18.29.238.0/23 *[BGP/170] 1d 19:57:28, localpref 90, from
>> 103.97.52.2
>>>AS path: 4637 3257 29909 16532 16532 16532
>>> 16532
>> I, validation-state: unverified
>>> --
>>> -
>>> Alain Hebertaheb...@pubnix.net
>>> PubNIX Inc.
>>> 50 boul. St-Charles
>>> P.O. Box 26770 Beaconsfield, Quebec H9W 

Re: Juniper BGP Convergence Time

[Phil Lavin]

Ask if they will configure BFD for you. I’ve not found many transit providers 
that will, but it’s worth a shot and it will lower failure detection to circa 1 
second.


> On 16 May 2018, at 17:49, Adam Kajtar  wrote:
> 
> I could use static routes but I noticed since I moved to full routes I have
> had a lot fewer customer complaints about latency(especially when it comes
> to Voice and VPN traffic).
> 
> I wasn't using per-packet load balancing. I believe juniper default is per
> IP.
> 
> My timers are as follows
> Active Holdtime: 90
> Keepalive Interval: 30
> 
> Would I be correct in thinking I need to contact my ISP to lower these
> values?
> 
> An interesting note is when I had both ISPs connected into a single MX104
> the failover was just a few seconds.
> 
> Thanks again.
> 
> 
> 
>> On Tue, May 15, 2018 at 8:42 PM Ben Cannon  wrote:
>> 
>> Have you checked your timeouts ?
>> 
>> -Ben
>> 
>>> On May 15, 2018, at 4:09 PM, Kaiser, Erich  wrote:
>>> 
>>> Do you need full routes?  What about just a default route from BGP?
>>> 
>>> Erich Kaiser
>>> The Fusion Network
>>> er...@gotfusion.net
>>> Office: 815-570-3101
>>> 
>>> 
>>> 
>>> 
 On Tue, May 15, 2018 at 5:38 PM, Aaron Gould  wrote:
 
 You sure it doesn't have something to do with 60 seconds * 3 = 180 secs
>> of
 BGP neighbor Time out before it believes neighbor is dead and remove
>> routes
 to that neighbor?
 
 Aaron
 
> On May 15, 2018, at 9:10 AM, Adam Kajtar 
 wrote:
> 
> Hello:
> 
> I'm running two Juniper MX104s. Each MX has 1 ISP connected running
> BGP(full routes). iBGP is running between the routers via a two port
>> 20G
> lag. When one of the ISPs fails, it can take upwards of 2 minutes for
> traffic to start flowing correctly. The router has the correct route in
 the
> routing table, but it doesn't install it in the forwarding table for
>> the
> full two mins.
> 
> I have a few questions if anyone could answer them.
> 
> - What would a usual convergence time be for this setup?
> - Is there anything I could do speed this process up? (I tried
 Multipath)
> - Any tips and tricks would be much appreciated
> 
> Thanks in Advance
> --
> Adam Kajtar
> Systems Administrator
> City of Wadsworth
> akaj...@wadsworthcity.org
> -
> http://www.wadsworthcity.com
> 
> Facebook * |* Twitter
>  *|* Instagram
>  *|* YouTube
> 
 
 
>> 
> 
> 
> -- 
> Adam Kajtar
> Systems Administrator, Safety Services
> City of Wadsworth
> Office 330.335.2865
> Cell 330.485.6510
> akaj...@wadsworthcity.org
> -
> http://www.wadsworthcity.com
> 
> Facebook * |* Twitter
>  *|* Instagram
>  *|* YouTube
>