Re: lots of internet starting at ~3 a.m. cst

[Aaron Gould]

understood.  today's ~3:00 a.m traffic increase, is similar to what I 
saw 1/23/2024... nice of them to at least start it during lowest use 
time of the day.


https://mailman.nanog.org/pipermail/nanog/2024-January/224671.html

-Aaron



On 7/23/2024 12:33 PM, Tom Beecher wrote:

No, because there's no set schedule for these things.

Some publishers / consoles get content/patches up in advance to allow 
user to pre-load. This can smooth out the bandwidth hits, but only if 
the users enable the pre-load feature. Many don't. Other publishers 
just enable the updates at once for everybody, and you see a big spike 
when they do.


The time of day is also not always the same either.

On Tue, Jul 23, 2024 at 10:09 AM Aaron Gould  wrote:

thanks Peter, et al.  Is there some sort of website, traffic
stats, gaming update schedule page for me to proactively see
if/when this type of thing will occur?  I mean, this is a
significant uptick on all 3 of my internet uplinks... would be
nice to know beforehand

-Aaron


On 7/23/2024 9:02 AM, Peter Potvin wrote:

Considering at least the top 4 IPs you've listed are CDN
providers (Edgio and Fastly), this definitely sounds like game
updates or something of the sort. Nothing unusual really for 3am
on a weekday morning especially for an eyeball network.

Kind regards,
Peter


On Tue, Jul 23, 2024 at 9:57 AM Aaron Gould  wrote:

Thanks Jason, updates for what?  I was hoping any other eyeball network operators 
may have been seeing "lots of internet" usage like me and may be able to share 
what they know.  I'm always suspicious about the typical game update that tends to cause 
something like this.  someone unicasted me a response that it might be related to F1 
Manager 2024 for video game consoles released today
  


i grabbed my netflow data for the last few hours for any source ip that 
has sent more than 5 GB of data...

Top 100 Src IP Addr ordered by bytes:
Date first seen  Duration Proto   Src IP Addr    Flows(%)   
  Packets(%)   Bytes(%) pps  bps   bpp
2024-07-23 04:37:29.600 14860.800 any  93.184.215.240   557558( 
0.6)  108.5 M( 9.2)  162.6 G(12.9) 7302   87.5 M  1497
2024-07-23 04:42:07.616 14577.664 any    72.21.81.240   361240( 
0.4)   64.5 M( 5.5)   96.6 G( 7.7) 4426   53.0 M  1497
2024-07-23 06:52:08.192  6781.696 any 151.101.162.172   270298( 
0.3)   36.9 M( 3.1)   55.0 G( 4.4) 5445   64.9 M  1489
2024-07-23 04:23:08.160 15722.240 any 151.101.150.172    1.0 M( 
1.0)   31.2 M( 2.6)   46.4 G( 3.7) 1985   23.6 M  1486
2024-07-23 04:42:47.552 14542.592 any  146.75.106.172   238287( 
0.2)   22.9 M( 1.9)   33.9 G( 2.7) 1575   18.7 M  1480
2024-07-23 03:40:11.776 18298.880 any 199.232.214.172   135793( 
0.1)   19.1 M( 1.6)   28.4 G( 2.3) 1043   12.4 M  1486
2024-07-23 03:48:59.392 17769.728 any 199.232.210.172   135811( 
0.1)   18.8 M( 1.6)   27.9 G( 2.2) 1057   12.6 M  1486
2024-07-23 04:22:05.184 15783.680 any  199.232.70.172   923350( 
0.9)   18.5 M( 1.6)   27.5 G( 2.2) 1169   14.0 M  1491
2024-07-23 04:36:40.704 14909.696 any   146.75.42.172   138813( 
0.1)    5.6 M( 0.5)    8.3 G( 0.7)  374    4.5 M  1492
2024-07-22 22:50:26.048 35683.840 any  199.232.70.252   112323( 
0.1)    5.4 M( 0.5)    8.0 G( 0.6)  150    1.8 M  1485
2024-07-23 06:52:06.656  6783.488 any   146.75.10.172    74859( 
0.1)    4.6 M( 0.4)    6.8 G( 0.5)  676    8.1 M  1489
2024-07-23 02:38:16.704 22011.392 any 151.101.160.204 6114( 
0.0)    4.1 M( 0.3)    6.2 G( 0.5)  187    2.2 M  1489
2024-07-23 01:34:17.728 25851.136 any 151.101.161.190    44001( 
0.0)    4.1 M( 0.3)    6.1 G( 0.5)  157    1.9 M  1489
2024-07-22 22:17:50.464 37638.912 any 199.232.154.252    69420( 
0.1)    3.6 M( 0.3)    5.4 G( 0.4)   96    1.1 M  1483

-Aaron



On 7/23/2024 8:19 AM, Peter Potvin wrote:

Do you have /any/ sort of additional information about this?
Which source and destination ASNs, how much "a lot" is, etc?
This sounds like a typical game update release cycle though
without any information not a whole lot of networks would be
able to confirm anything.

Kind regards,
Peter


On Tue, Jul 23, 2024 at 9:07 AM Aaron Gould
 wrote:

Anyone else see a lot of Internet traffic starting at 3
a.m. and
continuing even now?  Seems to be spiky tcp.

    -- 
-Aaron


    -- 
-Aaron


-- 
    -Aaron



--
-Aaron

Re: lots of internet starting at ~3 a.m. cst

[Aaron Gould]

thanks Peter, et al.  Is there some sort of website, traffic stats, 
gaming update schedule page for me to proactively see if/when this type 
of thing will occur?  I mean, this is a significant uptick on all 3 of 
my internet uplinks... would be nice to know beforehand


-Aaron


On 7/23/2024 9:02 AM, Peter Potvin wrote:
Considering at least the top 4 IPs you've listed are CDN providers 
(Edgio and Fastly), this definitely sounds like game updates or 
something of the sort. Nothing unusual really for 3am on a weekday 
morning especially for an eyeball network.


Kind regards,
Peter


On Tue, Jul 23, 2024 at 9:57 AM Aaron Gould  wrote:

Thanks Jason, updates for what?  I was hoping any other eyeball network operators may 
have been seeing "lots of internet" usage like me and may be able to share what 
they know.  I'm always suspicious about the typical game update that tends to cause 
something like this.  someone unicasted me a response that it might be related to F1 
Manager 2024 for video game consoles released today
  


i grabbed my netflow data for the last few hours for any source ip that has 
sent more than 5 GB of data...

Top 100 Src IP Addr ordered by bytes:
Date first seen  Duration Proto   Src IP Addr    Flows(%) 
Packets(%)   Bytes(%) pps  bps   bpp
2024-07-23 04:37:29.600 14860.800 any  93.184.215.240   557558( 0.6)  
108.5 M( 9.2)  162.6 G(12.9) 7302   87.5 M  1497
2024-07-23 04:42:07.616 14577.664 any    72.21.81.240   361240( 0.4)   
64.5 M( 5.5)   96.6 G( 7.7) 4426   53.0 M  1497
2024-07-23 06:52:08.192  6781.696 any 151.101.162.172   270298( 0.3)   
36.9 M( 3.1)   55.0 G( 4.4) 5445   64.9 M  1489
2024-07-23 04:23:08.160 15722.240 any 151.101.150.172    1.0 M( 1.0)   
31.2 M( 2.6)   46.4 G( 3.7) 1985   23.6 M  1486
2024-07-23 04:42:47.552 14542.592 any  146.75.106.172   238287( 0.2)   
22.9 M( 1.9)   33.9 G( 2.7) 1575   18.7 M  1480
2024-07-23 03:40:11.776 18298.880 any 199.232.214.172   135793( 0.1)   
19.1 M( 1.6)   28.4 G( 2.3) 1043   12.4 M  1486
2024-07-23 03:48:59.392 17769.728 any 199.232.210.172   135811( 0.1)   
18.8 M( 1.6)   27.9 G( 2.2) 1057   12.6 M  1486
2024-07-23 04:22:05.184 15783.680 any  199.232.70.172   923350( 0.9)   
18.5 M( 1.6)   27.5 G( 2.2) 1169   14.0 M  1491
2024-07-23 04:36:40.704 14909.696 any   146.75.42.172   138813( 0.1)    
5.6 M( 0.5)    8.3 G( 0.7)  374    4.5 M  1492
2024-07-22 22:50:26.048 35683.840 any  199.232.70.252   112323( 0.1)    
5.4 M( 0.5)    8.0 G( 0.6)  150    1.8 M  1485
2024-07-23 06:52:06.656  6783.488 any   146.75.10.172    74859( 0.1)    
4.6 M( 0.4)    6.8 G( 0.5)  676    8.1 M  1489
2024-07-23 02:38:16.704 22011.392 any 151.101.160.204 6114( 0.0)    
4.1 M( 0.3)    6.2 G( 0.5)  187    2.2 M  1489
2024-07-23 01:34:17.728 25851.136 any 151.101.161.190    44001( 0.0)    
4.1 M( 0.3)    6.1 G( 0.5)  157    1.9 M  1489
2024-07-22 22:17:50.464 37638.912 any 199.232.154.252    69420( 0.1)    
3.6 M( 0.3)    5.4 G( 0.4)   96    1.1 M  1483

-Aaron



On 7/23/2024 8:19 AM, Peter Potvin wrote:

Do you have /any/ sort of additional information about this?
Which source and destination ASNs, how much "a lot" is, etc? This
sounds like a typical game update release cycle though without
any information not a whole lot of networks would be able to
confirm anything.

Kind regards,
Peter


On Tue, Jul 23, 2024 at 9:07 AM Aaron Gould  wrote:

Anyone else see a lot of Internet traffic starting at 3 a.m. and
continuing even now?  Seems to be spiky tcp.

    -- 
-Aaron


-- 
    -Aaron



--
-Aaron

Re: lots of internet starting at ~3 a.m. cst

[Aaron Gould]

Thanks Jason, updates for what?  I was hoping any other eyeball network operators may 
have been seeing "lots of internet" usage like me and may be able to share what 
they know.  I'm always suspicious about the typical game update that tends to cause 
something like this.  someone unicasted me a response that it might be related to F1 
Manager 2024 for video game consoles released today
 


i grabbed my netflow data for the last few hours for any source ip that has 
sent more than 5 GB of data...

Top 100 Src IP Addr ordered by bytes:
Date first seen  Duration Proto   Src IP Addr    Flows(%) 
Packets(%)   Bytes(%) pps  bps   bpp
2024-07-23 04:37:29.600 14860.800 any  93.184.215.240   557558( 0.6)  108.5 
M( 9.2)  162.6 G(12.9) 7302   87.5 M  1497
2024-07-23 04:42:07.616 14577.664 any    72.21.81.240   361240( 0.4)   64.5 
M( 5.5)   96.6 G( 7.7) 4426   53.0 M  1497
2024-07-23 06:52:08.192  6781.696 any 151.101.162.172   270298( 0.3)   36.9 
M( 3.1)   55.0 G( 4.4) 5445   64.9 M  1489
2024-07-23 04:23:08.160 15722.240 any 151.101.150.172    1.0 M( 1.0)   31.2 
M( 2.6)   46.4 G( 3.7) 1985   23.6 M  1486
2024-07-23 04:42:47.552 14542.592 any  146.75.106.172   238287( 0.2)   22.9 
M( 1.9)   33.9 G( 2.7) 1575   18.7 M  1480
2024-07-23 03:40:11.776 18298.880 any 199.232.214.172   135793( 0.1)   19.1 
M( 1.6)   28.4 G( 2.3) 1043   12.4 M  1486
2024-07-23 03:48:59.392 17769.728 any 199.232.210.172   135811( 0.1)   18.8 
M( 1.6)   27.9 G( 2.2) 1057   12.6 M  1486
2024-07-23 04:22:05.184 15783.680 any  199.232.70.172   923350( 0.9)   18.5 
M( 1.6)   27.5 G( 2.2) 1169   14.0 M  1491
2024-07-23 04:36:40.704 14909.696 any   146.75.42.172   138813( 0.1)    5.6 
M( 0.5)    8.3 G( 0.7)  374    4.5 M  1492
2024-07-22 22:50:26.048 35683.840 any  199.232.70.252   112323( 0.1)    5.4 
M( 0.5)    8.0 G( 0.6)  150    1.8 M  1485
2024-07-23 06:52:06.656  6783.488 any   146.75.10.172    74859( 0.1)    4.6 
M( 0.4)    6.8 G( 0.5)  676    8.1 M  1489
2024-07-23 02:38:16.704 22011.392 any 151.101.160.204 6114( 0.0)    4.1 
M( 0.3)    6.2 G( 0.5)  187    2.2 M  1489
2024-07-23 01:34:17.728 25851.136 any 151.101.161.190    44001( 0.0)    4.1 
M( 0.3)    6.1 G( 0.5)  157    1.9 M  1489
2024-07-22 22:17:50.464 37638.912 any 199.232.154.252    69420( 0.1)    3.6 
M( 0.3)    5.4 G( 0.4)   96    1.1 M  1483

-Aaron



On 7/23/2024 8:19 AM, Peter Potvin wrote:
Do you have /any/ sort of additional information about this? Which 
source and destination ASNs, how much "a lot" is, etc? This sounds 
like a typical game update release cycle though without any 
information not a whole lot of networks would be able to confirm 
anything.


Kind regards,
Peter


On Tue, Jul 23, 2024 at 9:07 AM Aaron Gould  wrote:

Anyone else see a lot of Internet traffic starting at 3 a.m. and
continuing even now?  Seems to be spiky tcp.

-- 
    -Aaron



--
-Aaron

lots of internet starting at ~3 a.m. cst

[Aaron Gould]

Anyone else see a lot of Internet traffic starting at 3 a.m. and 
continuing even now?  Seems to be spiky tcp.


--
-Aaron

Re: Current diameter of the Internet?

[Aaron Groom]

If worst-case is an option, there are some interesting routing policies
between certain places.

One example is a Australia to China--take Perth to Chongqing as an
example.  They're at about the same longitude, but RTT is routinely
greater than 500 ms.  Packets travel to Singapore, then cross the entire
Pacific ocean to the west coast of the US, and back, before making it
back to China.

Try it for yourself from this Perth-based looking glass[1].

PING 117.151.152.239 (117.151.152.239) 56(84) bytes of data.
64 bytes from 117.151.152.239: icmp_seq=1 ttl=236 time=511 ms
64 bytes from 117.151.152.239: icmp_seq=2 ttl=236 time=522 ms

traceroute to 117.151.152.239 (117.151.152.239), 30 hops max, 60 byte packets
 1  172.18.0.1 (172.18.0.1)  0.021 ms  0.004 ms  0.004 ms
 2  45.248.78.65 (45.248.78.65)  0.611 ms  0.696 ms  0.777 ms
 3  45.248.78.139 (45.248.78.139)  0.678 ms  0.818 ms  0.959 ms
 4  core.p1.wa.hostuniversal.com.au (103.216.222.7)  0.145 ms  0.165 ms  0.195 
ms
 5  be6745.201.ccr51.per01.atlas.cogentco.com (154.18.100.65)  0.468 ms  0.418 
ms  0.496 ms
 6  be2428.ccr31.sin01.atlas.cogentco.com (154.54.88.138)  46.625 ms  46.659 ms 
 46.691 ms
 7  be2913.ccr41.lax04.atlas.cogentco.com (154.54.27.54)  218.213 ms  237.901 
ms  237.959 ms
[...]

1: https://perth-lg.ransomit.com.au/

Sean Donelan  writes:

> What is the current estimated diameter of the Internet?
>
> Maximum (worst-case) RTT edge-to-edge?
>
> Most public latency data is now edge-to-cloud, not edge-to-edge.
> Cloud engineers have done a great job, and edge-to-cloud less than
> 1-sec RTT.
>
> Where have the long-slow pipes gone?
>
> https://www.cloudping.co/grid
>
> https://learn.microsoft.com/en-us/azure/networking/azure-network-latency?tabs=Americas%2CWestUS
>
> https://www.verizon.com/business/terms/latency/

Re: charging for config changess

[Aaron C. de Bruyn via NANOG]

On Mon, Jul 1, 2024 at 11:01 AM heasley  wrote:

> Sun, Jun 30, 2024 at 02:17:12PM -0700, Randy Bush:
> Many of those things could be automated via a customer portal.
>

Oh totally.  Want to update your reverse DNS record?  Just put in a credit
card number and *bam* $250 charge to your card. ;)

Re: charging for config changess

[aaron]

From experience, I can imagine that Arelion has a very small subset of 
customers that are blowing up their tech support for changes like these 
constantly.  I'm sure it's a punitive measure to deter these guys.


Aaron


On 30.06.2024 20:33, Tim Burke wrote:

First I've heard of a provider doing it... and we do business with 
3356, the one

carrier I'd expect to do something like this :-)

Might just be me, but I rarely have to have config changes done on 
circuits
after provisioning, short of enabling dual stack bgp on a circuit that 
didn't
have it previously, or if a provider did something silly with your 
config after

provisioning/acceptance like send you a default route all of a sudden.

Despite that, I know there are lots of people that can't decide on how 
they
want to do things, or refuse to use and/or don't understand things like 
IRR.
I don't do anything with 1299 (yet), but I could potentially see this 
as a
"PITA surcharge" to discourage people from being unable to make their 
minds up...
surely they would waive it for clueful customers who are making a 
reasonable

quantity of changes.

On Jun 30, 2024, at 4:17 PM, Randy Bush  wrote: has 
charging for config changes a la 
https://www.arelion.com/customer-excellence/customer-support/online-technical-change-pricing 
[1] become common while i was not looking? admittedly, i have not 
looked for a long time. randy




Links:
--
[1] 
https://www.arelion.com/customer-excellence/customer-support/online-technical-change-pricing

Re: AS6762 Looking Glass Down

[Aaron Atac via NANOG]

Thanks, all!

-Aaron
Jun 26, 2024 at 5:26 AM by b...@benjojo.co.uk:

> Unsure what you are looking for (if it's just a general table view of
> Sparkle, or you are looking for a specific router etc) but there are
> *some* TI/Sparkle BGP tables (both v4 and v6) on the bgp.tools "super"
> looking glass https://bgp.tools/super-lg but I never managed to
> find/talk to someone at Sparkle to setup direct feeding sessions.
>
> RIPE Atlas can do the traceroute work load in the meantime?
>
> Assuming your target does not have PNIs or other peering with Telicom
> Italia domestic network, 3269, you can request RIPE Atlas traceroutes
> from there, otherwise AS272864 has atlas probes and seems to be
> entirely behind a upstream that is entirely behind TI.
>
> On Wed, 26 Jun 2024 at 08:42, Marcin Gondek  wrote:
>
>>
>> Hi
>>
>> Seems that do not support IPv6 :-/
>>
>> Thanks,
>>
>> --
>>
>> Marcin Gondek / Drixter
>> http://fido.e-utp.net/
>> AS56662
>> ____
>> Od: NANOG  w imieniu użytkownika 
>> Chris Welti via NANOG 
>> Wysłane: środa, 26 czerwca 2024 06:58
>> Do: Aaron Atac ; Nanog 
>> Temat: Re: AS6762 Looking Glass Down
>>
>> Hi Aaron,
>>
>> have a look at https://www.tisparkle.com/looking-glass
>>
>> Cheers,
>> Chris
>>
>> On 25.06.24 19:13, Aaron Atac via NANOG wrote:
>> > Hi,
>> >
>> > It seems the looking glass provided on peeringdb for AS6762 is down. 
>> > Anyone on the list know if there's an alternative link or when it might be 
>> > back up?
>> >
>> > https://www.peeringdb.com/net/31
>> >
>> > https://gambadilegno.noc.seabone.net/lg/
>> >
>> > Thanks,
>> > Aaron
>>

AS6762 Looking Glass Down

[Aaron Atac via NANOG]

Hi,

It seems the looking glass provided on peeringdb for AS6762 is down. Anyone on 
the list know if there's an alternative link or when it might be back up?

https://www.peeringdb.com/net/31

https://gambadilegno.noc.seabone.net/lg/

Thanks,
Aaron

Correcting national address databases?

[Aaron C. de Bruyn via NANOG]

I'm guessing someone in the community has experience dealing with this.

About 3 years ago my street got typo'd in some sort of national database of
addresses.  Two characters were transposed.  i.e. "Mian St" vs "Main St".

It's causing no end of issues with ordering online, pretty much every
shipper has picked up the bad address, and some of the mapping tools too.
Google and OSM appear to be the exceptions.

Any idea where to go to get this fixed?

-A

Re: Packet loss and latency between Akamai and NTT in Miami

[Aaron Atac via NANOG]

Responding off list.

-Aaron

May 17, 2024 at 12:42 PM by carlosm3...@gmail.com:

> Any contacts with either Akamai or NTT here ?
>
> This is kind of important as this is affecting three of our RPKI
> publication servers (servers which I have de-priorized in Route53 to
> prevent any issues for RPs)
>
> I have a ticket open with Akamai but I'm not directly an NTT customer
> so any help is appreciated.
>
> A sample MTR report, see between hops 7 and 8. Funnily enough this is
> periodic and has a cycle of between 15 and 25 minutes.
>
> %% START MTR TCP IPV4 en 20240517-16:10
> Start: 2024-05-17T16:10:01+
> HOST: rpki-fe-45-79-203-193.rrdp. Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1.|-- 10.204.6.210.0%200.5   0.2   0.1   0.5   0.1
>  2.|-- 10.204.35.59   0.0%200.3   0.3   0.2   0.6   0.1
>  10.204.35.60
>  3.|-- 10.204.64.38   0.0%200.3   0.3   0.2   0.6   0.1
>  10.204.64.37
>  4.|-- lo0-0.gw3.atl1.us.linode.  0.0%20   19.9   4.1   0.4  19.9   6.1
>  lo0-0.gw4.atl1.us.linode.com
>  5.|-- ae45.r12.atl01.ien.netarc  0.0%200.5   0.5   0.4   0.7   0.1
>  ae48.r11.atl01.ien.netarch.akamai.com
>  6.|-- ae-41.a03.atlnga05.us.bb.  0.0%205.2   2.3   0.4  14.8   3.7
>  ae0.r11.atl01.ien.netarch.akamai.com
>  7.|-- ae-41.a03.atlnga05.us.bb.  0.0%203.3  11.4   0.9  38.5  11.0
>  ae-2.r25.atlnga05.us.bb.gin.ntt.net
>  8.|-- ae-1.r22.miamfl02.us.bb.g 20.0%20  7271. 1816.   0.8 7271. 3236.6
>  ae-2.r25.atlnga05.us.bb.gin.ntt.net
>  9.|-- ae-1.r22.miamfl02.us.bb.g  5.0%20  7201. 1684.  13.3 7256. 2970.2
>  ae-0.r23.miamfl02.us.bb.gin.ntt.net
>  10.|-- ae-11.a00.saplbr02.br.bb.  0.0%20  123.2 206.8  13.1 3035. 667.7
>  ae-0.r23.miamfl02.us.bb.gin.ntt.net
>  11.|-- ae1-1326.gw1.nu.registro.  0.0%20  129.2 125.2 116.7 139.5   5.8
>  ae-11.a00.saplbr02.br.bb.gin.ntt.net
>  12.|-- xe-0-1-2-0.core1.nu.regis  0.0%20  117.9 174.0 117.6 1133. 225.9
>  ae1-1326.gw1.nu.registro.br
>
> Thanks
>
> Carlos
>
> -- 
> --
> =
> Carlos M. Martinez-Cagnazzo
> http://cagnazzo.me
> =
>

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

Thanks.  What "all the ethernet control frame juju" might you be 
referring to?  I don't recall Ethernet, in and of itself, just sending 
stuff back and forth.  Does anyone know if this FEC stuff I see 
concurring is actually contained in Ethernet Frames?  If so, please send 
a link to show the ethernet frame structure as it pertains to this 400g 
fec stuff.  If so, I'd really like to know the header format, etc.


-Aaron

On 4/18/2024 1:17 PM, Tom Beecher wrote:

FEC is occurring at the PHY , below the PCS.

Even if you're not sending any traffic, all the ethernet control frame 
juju is still going back and forth, which FEC may have to correct.


I *think* (but not 100% sure) that for anything that by spec requires 
FEC, there is a default RS-FEC type that will be used, which *may* be 
able to be changed by the device. Could be fixed though, I honestly 
cannot remember.


On Thu, Apr 18, 2024 at 1:35 PM Aaron Gould  wrote:

Not to belabor this, but so interesting... I need a FEC-for-Dummies or 
FEC-for-IP/Ethernet-Engineers...

Shown below, my 400g interface with NO config at all... Interface has no 
traffic at all, no packets at all  BUT, lots of FEC hits.  Interesting this 
FEC-thing.  I'd love to have a fiber splitter and see if wireshark could read 
it and show me what FEC looks like...but something tells me i would need a 400g 
sniffer to read it, lol

It's like FEC (fec119 in this case) is this automatic thing running between 
interfaces (hardware i guess), with no protocols and nothing needed at all in 
order to function.

-Aaron

{master}
me@mx960> show configuration interfaces et-7/1/4 | display set

{master}
me@mx960>

{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
     Input packets : 0
     Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
     Bit errors 0
     Errored blocks 0
   Ethernet FEC statistics  Errors
     FEC Corrected Errors    28209
     FEC Uncorrected Errors  0
     FEC Corrected Errors Rate    2347
     FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
     Input packets : 0
     Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
     Bit errors 0
     Errored blocks 0
   Ethernet FEC statistics  Errors
     FEC Corrected Errors    45153
     FEC Uncorrected Errors  0
     FEC Corrected Errors Rate  29
     FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
     Input packets : 0
     Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
     Bit errors 0
     Errored blocks 0
   Ethernet FEC statistics  Errors
     FEC Corrected Errors    57339
     FEC Uncorrected Errors  0
     FEC Corrected Errors Rate    2378
     FEC Uncorrected Errors Rate 0

{master}
me@mx960>


On 4/18/2024 7:13 AM, Mark Tinka wrote:



On 4/17/24 23:24, Aaron Gould wrote:


Well JTAC just said that it seems ok, and that 400g is going to
show 4x more than 100g "This is due to having to synchronize
much more to support higher data."



We've seen the same between Juniper and Arista boxes in the same
rack running at 100G, despite cleaning fibres, swapping optics,
moving ports, moving line cards, e.t.c. TAC said it's a
non-issue, and to be expected, and shared the same KB's.

It's a b

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

Not to belabor this, but so interesting... I need a FEC-for-Dummies or 
FEC-for-IP/Ethernet-Engineers...

Shown below, my 400g interface with NO config at all... Interface has no 
traffic at all, no packets at all  BUT, lots of FEC hits.  Interesting this 
FEC-thing.  I'd love to have a fiber splitter and see if wireshark could read 
it and show me what FEC looks like...but something tells me i would need a 400g 
sniffer to read it, lol

It's like FEC (fec119 in this case) is this automatic thing running between 
interfaces (hardware i guess), with no protocols and nothing needed at all in 
order to function.

-Aaron

{master}
me@mx960> show configuration interfaces et-7/1/4 | display set

{master}
me@mx960>

{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
    Input packets : 0
    Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
  Input rate : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
    Bit errors 0
    Errored blocks 0
  Ethernet FEC statistics  Errors
    FEC Corrected Errors    28209
    FEC Uncorrected Errors  0
    FEC Corrected Errors Rate    2347
    FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
    Input packets : 0
    Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
  Input rate : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
    Bit errors 0
    Errored blocks 0
  Ethernet FEC statistics  Errors
    FEC Corrected Errors    45153
    FEC Uncorrected Errors  0
    FEC Corrected Errors Rate  29
    FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
    Input packets : 0
    Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
  Input rate : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
    Bit errors 0
    Errored blocks 0
  Ethernet FEC statistics  Errors
    FEC Corrected Errors    57339
    FEC Uncorrected Errors  0
    FEC Corrected Errors Rate    2378
    FEC Uncorrected Errors Rate 0

{master}
me@mx960>


On 4/18/2024 7:13 AM, Mark Tinka wrote:



On 4/17/24 23:24, Aaron Gould wrote:

Well JTAC just said that it seems ok, and that 400g is going to show 
4x more than 100g "This is due to having to synchronize much more to 
support higher data."




We've seen the same between Juniper and Arista boxes in the same rack 
running at 100G, despite cleaning fibres, swapping optics, moving 
ports, moving line cards, e.t.c. TAC said it's a non-issue, and to be 
expected, and shared the same KB's.


It's a bit disconcerting when you plot the data on your NMS, but it's 
not material.


Mark.


--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

Well JTAC just said that it seems ok, and that 400g is going to show 4x 
more than 100g "This is due to having to synchronize much more to 
support higher data."


-Aaron



On 4/17/2024 4:04 PM, Aaron Gould wrote:


Interesting, thanks all, the JTAC rep got back to me and also pretty 
much said it's not an issue and is expected... also, JTAC rep sited 2 
KB's, shown here, both using 100g as an example... question please, 
should I understand that this is also true about 400g, even though his 
KB's speak about 100g ?


KB77305
KB35145

https://supportportal.juniper.net/s/article/What-is-the-acceptable-rate-of-FEC-corrected-errors-for-100G-interface 

https://supportportal.juniper.net/s/article/PTX-FEC-corrected-errors-increasing-on-link-between-QSFP-100GBASE-SR4-740-058734-and-QSFP-100G-SR4-T2-740-061405?language=en_US 



-Aaron


On 4/17/2024 3:58 PM, Matt Erculiani wrote:
At some point, an error rate would exceed the ability of forward 
error correction (FEC) overhead to compensate, resulting in CRC 
errors. You're not seeing those so all is technically well.


It's not so much how many packets come in with errors that causes a 
problem, but what percentage of each packet is corrupted. The former 
is usually indicative of the latter though.


Just as Tom said, we're talking about a whole new animal than the NRZ 
we're used to inside the building. Long-haul and DCI folks deal with 
this stuff pretty regularly. The secret is keep everything clean and 
mind your bend radii. We won't get away with some of what we used to 
get away with.


-Matt

On Wed, Apr 17, 2024 at 1:49 PM Aaron Gould  wrote:

fec cliff?  is there a level of fec erros that i should be
worried about then?  not sure what you mean.

-Aaron

On 4/17/2024 2:46 PM, Matt Erculiani wrote:

I'm no TAC engineer, but the purpose of FEC is to take and
correct errors when the port is going so fast that errors are
simply inevitable. Working as Intended.

Easier (read: cheaper) to build in some error correction than
make the bits wiggle more reliably.

No idea if that rate of increment is alarming or not, but you've
not yet hit your FEC cliff so you appear to be fine.

-Matt

On Wed, Apr 17, 2024 at 1:40 PM Dominik Dobrowolski
 wrote:

Open a JTAC case,
That looks like a work for them


Kind Regards,
Dominik

W dniu śr., 17.04.2024 o 21:36 Aaron Gould 
napisał(a):

We recently added MPC10E-15C-MRATE cards to our MX960's to upgrade 
our core to 400g.  During initial testing of the 400g interface (400GBASE-FR4), 
I see constant FEC errors.  FEC is new to me.  Anyone know why this is 
occurring?  Shown below, is an interface with no traffic, but seeing constant 
FEC errors.  This is (2) MX960's cabled directly, no dwdm or anything between 
them... just a fiber patch cable.



{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep rror | refresh 2
---(refreshed at 2024-04-17 14:18:53 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors0
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   0
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:55 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 4302
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   8
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:57 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 8796
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 146
 FEC Uncorre

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

Interesting, thanks all, the JTAC rep got back to me and also pretty 
much said it's not an issue and is expected... also, JTAC rep sited 2 
KB's, shown here, both using 100g as an example... question please, 
should I understand that this is also true about 400g, even though his 
KB's speak about 100g ?


KB77305
KB35145

https://supportportal.juniper.net/s/article/What-is-the-acceptable-rate-of-FEC-corrected-errors-for-100G-interface 

https://supportportal.juniper.net/s/article/PTX-FEC-corrected-errors-increasing-on-link-between-QSFP-100GBASE-SR4-740-058734-and-QSFP-100G-SR4-T2-740-061405?language=en_US 



-Aaron


On 4/17/2024 3:58 PM, Matt Erculiani wrote:
At some point, an error rate would exceed the ability of forward error 
correction (FEC) overhead to compensate, resulting in CRC errors. 
You're not seeing those so all is technically well.


It's not so much how many packets come in with errors that causes a 
problem, but what percentage of each packet is corrupted. The former 
is usually indicative of the latter though.


Just as Tom said, we're talking about a whole new animal than the NRZ 
we're used to inside the building. Long-haul and DCI folks deal with 
this stuff pretty regularly. The secret is keep everything clean and 
mind your bend radii. We won't get away with some of what we used to 
get away with.


-Matt

On Wed, Apr 17, 2024 at 1:49 PM Aaron Gould  wrote:

fec cliff?  is there a level of fec erros that i should be worried
about then?  not sure what you mean.

-Aaron

On 4/17/2024 2:46 PM, Matt Erculiani wrote:

I'm no TAC engineer, but the purpose of FEC is to take and
correct errors when the port is going so fast that errors are
simply inevitable. Working as Intended.

Easier (read: cheaper) to build in some error correction than
make the bits wiggle more reliably.

No idea if that rate of increment is alarming or not, but you've
not yet hit your FEC cliff so you appear to be fine.

-Matt

On Wed, Apr 17, 2024 at 1:40 PM Dominik Dobrowolski
 wrote:

Open a JTAC case,
That looks like a work for them


Kind Regards,
Dominik

W dniu śr., 17.04.2024 o 21:36 Aaron Gould 
napisał(a):

We recently added MPC10E-15C-MRATE cards to our MX960's to upgrade 
our core to 400g.  During initial testing of the 400g interface (400GBASE-FR4), 
I see constant FEC errors.  FEC is new to me.  Anyone know why this is 
occurring?  Shown below, is an interface with no traffic, but seeing constant 
FEC errors.  This is (2) MX960's cabled directly, no dwdm or anything between 
them... just a fiber patch cable.



{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep rror | refresh 2
---(refreshed at 2024-04-17 14:18:53 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors0
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   0
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:55 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 4302
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   8
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:57 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 8796
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 146
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:59 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, 

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

Thanks Joe and Schylar, that's reassuring.  Tom, yes, I believe fec is 
required for 400g as you see fec119 listed in that output... and i 
understand you can't (or perhaps shouldn't) change it.


-Aaron

On 4/17/2024 2:43 PM, Joe Antkowiak wrote:

Corrected FEC errors are pretty normal for 400G FR4



On Wednesday, April 17th, 2024 at 3:36 PM, Aaron Gould 
 wrote:

We recently added MPC10E-15C-MRATE cards to our MX960's to upgrade our core to 
400g.  During initial testing of the 400g interface (400GBASE-FR4), I see 
constant FEC errors.  FEC is new to me.  Anyone know why this is occurring?  
Shown below, is an interface with no traffic, but seeing constant FEC errors.  
This is (2) MX960's cabled directly, no dwdm or anything between them... just a 
fiber patch cable.



{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep rror | refresh 2
---(refreshed at 2024-04-17 14:18:53 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors0
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   0
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:55 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 4302
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   8
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:57 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 8796
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 146
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:59 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors15582
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 111
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:19:01 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors20342
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 256
 FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4
Physical interface: et-7/1/4, Enabled, Physical link is Up
   Interface index: 226, SNMP ifIndex: 800
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
   Flow control: Enabled
   Pad to minimum frame size: Disabled
   Device flags   : Present Running
   Interface flags: SNMP-Traps Internal: 0x4000
   Link flags : None
   CoS queues : 8 supported, 8 maximum usable queues
   Schedulers : 0
   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)
   Active alarms  : None
   Active defects : None
   PCS statistics  Seconds
 Bit errors 0
 Errored blocks 0
   Ethernet FEC Mode  : FEC119
   Ethernet FEC statistics  Errors
 FEC Corrected Errors   801787
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate2054
 FEC Uncorrected Errors Rate 0
   Link Degrade :
 Link Monitoring   :  Disable
   Interface transmi

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

fec cliff?  is there a level of fec erros that i should be worried about 
then?  not sure what you mean.


-Aaron

On 4/17/2024 2:46 PM, Matt Erculiani wrote:
I'm no TAC engineer, but the purpose of FEC is to take and correct 
errors when the port is going so fast that errors are 
simply inevitable. Working as Intended.


Easier (read: cheaper) to build in some error correction than make the 
bits wiggle more reliably.


No idea if that rate of increment is alarming or not, but you've not 
yet hit your FEC cliff so you appear to be fine.


-Matt

On Wed, Apr 17, 2024 at 1:40 PM Dominik Dobrowolski 
 wrote:


Open a JTAC case,
That looks like a work for them


Kind Regards,
Dominik

W dniu śr., 17.04.2024 o 21:36 Aaron Gould 
napisał(a):

We recently added MPC10E-15C-MRATE cards to our MX960's to upgrade our 
core to 400g.  During initial testing of the 400g interface (400GBASE-FR4), I 
see constant FEC errors.  FEC is new to me.  Anyone know why this is occurring? 
 Shown below, is an interface with no traffic, but seeing constant FEC errors.  
This is (2) MX960's cabled directly, no dwdm or anything between them... just a 
fiber patch cable.



{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep rror | refresh 2
---(refreshed at 2024-04-17 14:18:53 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors0
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   0
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:55 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 4302
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   8
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:57 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 8796
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 146
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:59 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors15582
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 111
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:19:01 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors20342
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 256
 FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4
Physical interface: et-7/1/4, Enabled, Physical link is Up
   Interface index: 226, SNMP ifIndex: 800
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
   Flow control: Enabled
   Pad to minimu

Re: constant FEC errors juniper mpc10e 400g

[Aaron Gould]

i did.  Usually my NANOG and J-NSP email list gets me a quicker solution 
than JTAC.


-Aaron

On 4/17/2024 2:37 PM, Dominik Dobrowolski wrote:

Open a JTAC case,
That looks like a work for them


Kind Regards,
Dominik

W dniu śr., 17.04.2024 o 21:36 Aaron Gould  napisał(a):

We recently added MPC10E-15C-MRATE cards to our MX960's to upgrade our core 
to 400g.  During initial testing of the 400g interface (400GBASE-FR4), I see 
constant FEC errors.  FEC is new to me.  Anyone know why this is occurring?  
Shown below, is an interface with no traffic, but seeing constant FEC errors.  
This is (2) MX960's cabled directly, no dwdm or anything between them... just a 
fiber patch cable.



{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep rror | refresh 2
---(refreshed at 2024-04-17 14:18:53 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors0
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   0
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:55 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 4302
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate   8
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:57 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors 8796
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 146
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:18:59 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors15582
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 111
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:19:01 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors20342
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 256
 FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4
Physical interface: et-7/1/4, Enabled, Physical link is Up
   Interface index: 226, SNMP ifIndex: 800
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
   Flow control: Enabled
   Pad to minimum frame size: Disabled
   Device flags   : Present Running
   Interface flags: SNMP-Traps Internal: 0x4000
   Link flags : None
   CoS queues : 8 supported, 8 maximum usable queues
   Schedulers : 0
   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)
   Active alarms  : None
   Active defects : None
   PCS statistics  Seconds
 Bit errors 0
 Errored blocks 0
   Ethernet FEC Mode  : FEC119
   Ethernet FEC statistics  Errors
 FEC Corrected Errors   801787
 

constant FEC errors juniper mpc10e 400g

[Aaron Gould]

30, Local: 10.10.10.77, Broadcast: 10.10.10.79

--
-Aaron

Re: Netskrt - ISP-colo CDN

[Aaron Gould]

I've had my dual-100g-connected Amazon ACEv2 caches for over a year 
now.  With my ~55,000 subs I saw every Thursday night for NFL/TNF usage 
at 15 gbps X2 (so 30 gbps total) and one day in late November 
(thanksgiving probably) I saw 25 gbps x2 (so 50 gbps) usage!


-Aaron

On 4/4/2024 6:08 PM, Paul Bradford wrote:
I have some on my network.  I don't think they populate content from 
their own cdn network, but it comes from Amazon.   interestingly for 
the NFL super bowl, while paramount+ streamed the game, on Amazon 
Prime Video you could "Watch super bowl on paramount+ Via Prime.". 
 that did actually drive users to using the netskrt caches.


They seem to work OK.  TNF in 6 months will tell us more.  :)



On Thu, Apr 4, 2024 at 6:14 PM John Stitt  wrote:

The website says they are part of the Streaming Video Technology
Alliance.

I wonder if this is a prepackaged Open Cache box.

https://opencaching.svta.org/

We also don’t appear to have had any traffic from them.  Not much
on the peeringdb for the USA ASN either.

BGP.tools shows they have upstreams with each ASN, and are on Ohio
IX with AS53471, but not really any peers anywhere.  Looks like
Cogent and Zayo for upstreams and only peer I see is AS1239
(Sprint Wireline (Cogent))

John Stitt

*From:*NANOG  *On
Behalf Of *Aaron Gould
*Sent:* Thursday, April 4, 2024 4:36 PM
*To:* Eric Dugas 
*Cc:* nanog@nanog.org
*Subject:* Re: Netskrt - ISP-colo CDN




You don't often get email from aar...@gvtc.com. Learn why this is
important <https://aka.ms/LearnAboutSenderIdentification>



Thanks... they told me it was free.

-Aaron

On 4/4/2024 4:12 PM, Eric Dugas wrote:

That name rang a bell so I looked up my emails.

They contacted me last year, they were claiming to be "working
with some of the major streaming brands, such as Amazon Prime
Video, to improve the quality of both VOD and live streaming
while also reducing the load on ISP networks such as your own.".

Based on my quick research, they have a few registered ASNs
(their peeringdb page <https://www.peeringdb.com/org/36226>)
with a few netblocks but I get 0 traffic from them (we're a
sizable eyeball network). Their origin network might still not
be ready but digging a little bit more, it seems they act as a
third-party video caching solution and not as an origin CDN so
in the end, they're really just trying to sell ISPs and other
types of customers their caching solutions.


Eric

    On Thu, Apr 4, 2024 at 4:00 PM Aaron Gould 
wrote:

Anyone out there using Netskrt CDN?  I mean, installed in
your network
for content delivery to your customers.  I understand
Netskrt provides
caching for some well known online video streaming
services... just
wondering if there are any network operators that have
worked with
Netskrt and deployed their caching servers in your
networks and what
have you thought about it?  What Internet uplink savings
are you seeing?

Netskrt - https://www.netskrt.io/


    -- 
    -Aaron


-- 


-Aaron

CAUTION:This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the
sender and know the content is safe. If you are not expecting this
message contact the sender directly via phone/text to verify.


--
-Aaron

Re: Netskrt - ISP-colo CDN

[Aaron Gould]

Thanks ... that svta caching sounds interesting.  i watched the 
presentation, but don't understand how it's used by ISP's that want to 
benefit from it.


-Aaron

On 4/4/2024 5:14 PM, John Stitt wrote:


The website says they are part of the Streaming Video Technology Alliance.

I wonder if this is a prepackaged Open Cache box.

https://opencaching.svta.org/

We also don’t appear to have had any traffic from them.  Not much on 
the peeringdb for the USA ASN either.


BGP.tools shows they have upstreams with each ASN, and are on Ohio IX 
with AS53471, but not really any peers anywhere.  Looks like Cogent 
and Zayo for upstreams and only peer I see is AS1239 (Sprint Wireline 
(Cogent))


John Stitt

*From:*NANOG  *On 
Behalf Of *Aaron Gould

*Sent:* Thursday, April 4, 2024 4:36 PM
*To:* Eric Dugas 
*Cc:* nanog@nanog.org
*Subject:* Re: Netskrt - ISP-colo CDN




You don't often get email from aar...@gvtc.com. Learn why this is 
important <https://aka.ms/LearnAboutSenderIdentification>




Thanks... they told me it was free.

-Aaron

On 4/4/2024 4:12 PM, Eric Dugas wrote:

That name rang a bell so I looked up my emails.

They contacted me last year, they were claiming to be "working
with some of the major streaming brands, such as Amazon Prime
Video, to improve the quality of both VOD and live streaming while
also reducing the load on ISP networks such as your own.".

Based on my quick research, they have a few registered ASNs (their
peeringdb page <https://www.peeringdb.com/org/36226>) with a few
netblocks but I get 0 traffic from them (we're a sizable eyeball
network). Their origin network might still not be ready but
digging a little bit more, it seems they act as a third-party
video caching solution and not as an origin CDN so in the end,
they're really just trying to sell ISPs and other types of
customers their caching solutions.


Eric

On Thu, Apr 4, 2024 at 4:00 PM Aaron Gould  wrote:

Anyone out there using Netskrt CDN?  I mean, installed in your
network
for content delivery to your customers.  I understand Netskrt
provides
caching for some well known online video streaming services...
just
wondering if there are any network operators that have worked
with
Netskrt and deployed their caching servers in your networks
and what
have you thought about it?  What Internet uplink savings are
you seeing?

Netskrt - https://www.netskrt.io/


-- 
-Aaron


--
-Aaron

CAUTION:This email originated from outside of the organization. Do not 
click links or open attachments unless you recognize the sender and 
know the content is safe. If you are not expecting this message 
contact the sender directly via phone/text to verify.



--
-Aaron

Re: Netskrt - ISP-colo CDN

[Aaron Gould]

Thanks... they told me it was free.

-Aaron

On 4/4/2024 4:12 PM, Eric Dugas wrote:

That name rang a bell so I looked up my emails.

They contacted me last year, they were claiming to be "working with 
some of the major streaming brands, such as Amazon Prime Video, to 
improve the quality of both VOD and live streaming while also reducing 
the load on ISP networks such as your own.".


Based on my quick research, they have a few registered ASNs (their 
peeringdb page <https://www.peeringdb.com/org/36226>) with a few 
netblocks but I get 0 traffic from them (we're a sizable eyeball 
network). Their origin network might still not be ready but digging a 
little bit more, it seems they act as a third-party video caching 
solution and not as an origin CDN so in the end, they're really just 
trying to sell ISPs and other types of customers their caching solutions.


Eric

On Thu, Apr 4, 2024 at 4:00 PM Aaron Gould  wrote:

Anyone out there using Netskrt CDN?  I mean, installed in your
network
for content delivery to your customers.  I understand Netskrt
provides
caching for some well known online video streaming services... just
wondering if there are any network operators that have worked with
Netskrt and deployed their caching servers in your networks and what
have you thought about it?  What Internet uplink savings are you
seeing?

Netskrt - https://www.netskrt.io/


    -- 
    -Aaron



--
-Aaron

Netskrt - ISP-colo CDN

[Aaron Gould]

Anyone out there using Netskrt CDN?  I mean, installed in your network 
for content delivery to your customers.  I understand Netskrt provides 
caching for some well known online video streaming services... just 
wondering if there are any network operators that have worked with 
Netskrt and deployed their caching servers in your networks and what 
have you thought about it?  What Internet uplink savings are you seeing?


Netskrt - https://www.netskrt.io/


--
-Aaron

Re: AWS Web Application Firewall blocks ISP ranges?

[Aaron Wendel]

Yes.

our network is a mix of content and eyeballs and they listed the whole 
thing.  This has prevented the local school district from using their 
text to speech application (for their deaf students) as well as others.  
This has also affected the local library and residences.


It's a PITA.

Thanks Amazon.

Aaron


On 3/21/2024 12:16 PM, Jonathan Kalbfeld via NANOG wrote:

Hi All,

I just became aware that AWS has a list of hosting IP providers and 
that list is blocked by their WAF? (!?!?).  None of my VM or colo 
customers can reach anything in AWS, such as Docker, Twilio, etc.  I 
confirmed through source routing that when I access it using one of my 
peering partners as a source IP it is reachable, but using one of my 
net blocks, it is not reachable and times out.  Checked all of my 
routing tables and those AWS blocks are definitely visible.  Also 
confirmed from looking glass that my IP ranges are showing up.


Has anyone else encountered that? If so, is there a way to get removed 
from that list? I have a very curated list of clients and I know all 
of them personally and none of them have been abusing AWS, so I was 
wondering if it was some kind of blanket ban?


If you're internal to AWS, my ASN is 54380, IP ranges affected are 
199.33.244.0/24, 199.79.202.0/24, 199.188.96.0/22, 45.59.144.0/22 and 
206.197.110.0/24


Feel free to reach out off-list.

Thanks,

Jonathan Kalbfeld

Jonathan Kalbfeld

office: +1 310 317 7933 
fax: +1 310 317 7901 
home: +1 310 317 7909 
mobile: +1 310 227 1662 

ThoughtWave Technologies, Inc.
Studio City, CA 91604
https://thoughtwave.com

View our network at
https://bgp.he.net/AS54380

+1 844 42-LINUX



--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
http://www.wholesaleinternet.com
aa...@wholesaleinternet.com

Re: Why are paper LOAs still used?

[Aaron Wendel]

I don't have any examples of anyone still using paper LOAs except for 
Cogent.


Aaron


On 2/26/2024 12:57 PM, Seth Mattinen via NANOG wrote:
Why do companies still insist on, or deploy new systems that rely on 
paper LOA for IP and ASN resources? How can this be considered more 
trustworthy than RIR based IRR records?


And I'm not even talking about old companies, I have a situation right 
now where a VPS provider I'm using will no longer use IRR and only 
accepts new paper LOAs. In the year 2024. I don't understand how 
anyone can go backwards like that.


~Seth

Seeking Contact From AS6079

[Aaron Atac via NANOG]

Hi,

Appreciate if someone from AS6079 could reach me off-list.

Thanks,
Aaron

edgecast - lots of traffic at ~3:00 a.m.

[Aaron Gould]

Anyone else see a lot of traffic inbound from the Internet last night 
(early this morning) at ~3:00 a.m. central time?  I see an IP Address, 
(93.184.215.240 - EdgeCast), which I think is EdgIO (fka limelight).  
Any idea what this is related to? (something tells me it's a game update)


--
-Aaron

Re: Fiber/OSP Technician Training and Apprenticeship Programs

[Aaron Axvig via NANOG]

On 2023-11-16 2:51 pm, Rhys Barrie via NANOG wrote:

Hey all,

I've recently been working with our county's broadband task force,
investigating the expansion and equity of broadband networks on a
local and state level. Through that, it's become clear that there's a
painful shortage of fiber / outside plant technicians in the state of
Michigan (if not nation-wide) in order to fulfill the workforce
requirements of maintaining the current broadband fiber infrastructure
in the state, much less to fuel fiber expansion, and especially in
rural areas. There appear to be few options for training the required
workforce, especially outside of the large enterprises that have the
resources to run their own internal programs, and small (or even
mid-sized) ISPs seem to be left with predominantly informal
person-to-person transfer of internal knowledge, assuming that they
have the required internal knowledge in the first place. This need for
a qualified workforce is exacerbated in the face of the multitude of
state and federal programs to encourage broadband internet expansion
and equity, such as the upcoming $42.5 billion in BEAD grant funding
and corresponding construction starting in ~12-18 months state- and
nation-wide.

As a result, our workforce development team over here at Mott
Community College (Genesee County, MI) is working to develop a fiber /
outside plant training and apprenticeship program in order to help
address this shortage of qualified personnel and training options at a
local and state level. We're looking for some industry contacts that
would be interested in collaborating with us to establish high-level
requirements regarding what skills need to be taught to prospective
fiber / outside plant technicians, what qualifications trainees should
have after completion in order to fulfill current workforce demands,
and to otherwise provide input in sketching out a high-level
curriculum. We're looking for feedback from a wide cross-section of
industry stakeholders -- large enterprise backbone transit providers,
rural residential ISPs, fiber co-ops and municipal networks,
operations and outside plant managers, etc. -- in order to determine
what the industry wants and needs, and how the entire community
college system can help meet those needs.

If anyone thinks that they have valuable input to provide regarding
these workforce requirements, or knows the right people to talk to,
please reach out and let me know!

Rhys Barrie (He/Him)
Network Engineer - Mott Community College
Member - Genesee County Broadband Task Force
(810) 762-0030 | rhys.bar...@mcc.edu | https://mcc.edu/


The Fiber Optic Association may be of interest to you. "The FOA is an 
international non-profit educational association that is chartered to 
promote professionalism in fiber optics through education, certification 
and standards."


https://www.thefoa.org/

I read a couple of their e-books to self-teach myself some fiber 
knowledge circa 2016 and it was good material.

ipv6 address management - documentation

[Aaron Gould]

For years I've used an MS Excel spreadsheet to manage my IPv4 
addresses.  IPv6 is going to be maddening to manage in a spreadsheet.  
What does everyone use for their IPv6 address prefix management and 
documentation?  Are there open source tools/apps for this?


--
-Aaron

MCC (Microsoft Connected Cache for ISP)

[Aaron Gould]

Is MCC for ISP comparable to other well-known CDN's, like Facebook FNA, 
Netflix OCA, etc?


Anyone have any experience with MCC in an ISP environment, and do you 
see much bandwidth savings with it?


https://learn.microsoft.com/en-us/windows/deployment/do/mcc-isp


--
-Aaron

Re: 165 Halsey recurring power issues

[Aaron Wendel]

I toured The Planet years ago in Dallas and was told by the sales rep 
that A+B power was two circuits from the same PDU. :)


I consider A+B power to be two distinct feeds, separate utility 
entrances, separate generators, separate UPS', PDU's, etc.  Past that I 
consider things like firewall separation, rated chases and such to be 
customer specific requirements.


Aaron

On 10/23/2023 9:38 AM, Babak Pasdar wrote:

Hello,

I wanted to get some feedback as to what is considered standard A/B 
power setup when data centers sell redundant power.  It has always 
been my understanding that A/B power means individually unique and 
preferably alternate path connections to disparate UPS units.


A few months ago, 165 Halsey took us down for several hours. They 
claimed that a UPS failed causing this issue.  Our natural reaction 
was that we have A/B redundant power so a failed UPS on the A circuit 
should not take down the cabinet. Joe the facility manager claimed 
that industry standard A/B power means two circuits to the same UPS, 
which makes no sense to me.


They committed to move us to A/B power with redundant circuits to 
disparate UPS units.  However, we had a multi-hour outage again in 
that site this weekend. At first glance it seems to be the same problem.


We have checked with all of our other data center providers who have 
confirmed A/B power is in fact individually unique connections to 
disparate UPS units. 165 Halsey's definition of what constitutes 
redundant power seems unique. Why would anyone pay extra for a second 
connection to the same UPS?  However, I wanted to get feedback to see 
if I am taking crazy pills here 


None-the-less, we have lost all confidence in this facility.

Best Regards,

Babak

Re: transit and peering costs projections

[Aaron Wendel]

The issue in Houston is Dallas.

I reached out to 30-40 networks and 90% of them all said they just back haul to 
Dallas and have no interest in peering in Houston.  It’s a real hard town to 
get any traction in.  If you’re local and have some insight, I’d be super happy 
to talk to you. 

Aaron

> On Oct 14, 2023, at 8:48 PM, Tim Burke  wrote:
> 
> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
> good bit less than $900 on the wholesale market.
> 
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
> costing as much as transit in many cases after you factor in loop costs.
> 
> For example, in the Houston market (one of the largest and fastest growing 
> regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
> several thousand for a 100g wave, plus several thousand for a 100g port on 
> one of those major IXes. Or, a better option, we can get a 100g flat internet 
> transit for just a little bit more.
> 
> Fortunately, for us as an eyeball network, there are a good number of major 
> content networks that are allowing for private peering in markets like 
> Houston for just the cost of a cross connect and a QSFP if you’re in the 
> right DC, with Google and some others being the outliers.
> 
> So for now, we'll keep paying for transit to get to the others (since it’s 
> about as much as transporting IXP from Dallas), and hoping someone at Google 
> finally sees Houston as more than a third rate city hanging off of Dallas. 
> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
> peering to Kansas City. Yeah, I think the former is more likely. 
> 
> See y’all in San Diego this week,
> Tim
> 
>> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>> 
>> This set of trendlines was very interesting. Unfortunately the data
>> stops in 2015. Does anyone have more recent data?
>> 
>> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>> 
>> I believe a gbit circuit that an ISP can resell still runs at about
>> $900 - $1.4k (?) in the usa? How about elsewhere?
>> 
>> ...
>> 
>> I am under the impression that many IXPs remain very successful,
>> states without them suffer, and I also find the concept of doing micro
>> IXPs at the city level, appealing, and now achievable with cheap gear.
>> Finer grained cross connects between telco and ISP and IXP would lower
>> latencies across town quite hugely...
>> 
>> PS I hear ARIN is planning on dropping the price for, and bundling 3
>> BGP AS numbers at a time, as of the end of this year, also.
>> 
>> 
>> 
>> --
>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos

Re: U.S. test of national alerts on Oct. 4 at 2:20pm EDT (1820 UTC)

[Aaron Wendel]

I think this is what he was referring to:

https://en.wikipedia.org/wiki/2018_Hawaii_false_missile_alert

Apparently we don't "all remember".



On 10/4/2023 1:39 PM, Sean Donelan wrote:

On Wed, 4 Oct 2023, Sabri Berisha wrote:
Makes me wonder what I have to do to opt out of this. We all remember 
what happened in Hawaii.


Do you mean the 98 people (at least) who died due to the Maui Lahaina 
wildfires.  Seems like the same people who complain about the testing 
of public warning systems also complain when they don't get a warning 
about something that personally affected them.



Public warning systems are designed to get your attention, wake you 
up, interrupt what you are doing.


Nevertheless, I understand some people will remove the batteries from 
smoke alarms and turn off public alerts.

Re: U.S. test of national alerts on Oct. 4 at 2:20pm EDT (1820 UTC)

[Aaron de Bruyn via NANOG]

I was kinda surprised that none of my NOAA weather radios went off. I sorta 
assumed they'd be tied into the whole "national" alert setup.

Why interrupt cell phones, AM/FM radio stations, and TV stations, but exclude 
NOAA weather radios?

-A

On Sun Oct 1, 2023, 10:24 PM GMT, Sean Donelan  wrote:
>
> This year's test of the U.S. national emergency alert includes something
> for ISPs and network operators.
>
> The wireless portion of the national test is scheduled 2 minutes (2:18pm
> EDT or 1818 UTC) before the main broadcast test at 2:20. Mobile phones
> usually receive the alert about a minute later. Radio and TV will receive
> the national alert a few minutes after 2:20pm.
>
> iPhone iOS 17 added a new feature for Wireless Emergency Alerts. When iOS
> 17 iPhones get a wireless emergency alert (WEA), it will trigger a data
> network query for additional information. Its a small query and
> response, but there are a lot of iPhones making the query at the same
> time (I'm assuming Apple engineer's have built in some time skew).
>
> Apple has assured FEMA that Apple's CDN and servers will be able to handle
> the triggered load.
>
> The iOS 17 triggered query will either be a tiny blip in the network
> graphs around 2:18pm to 2:22pm which no one will notice, or some CDNs and
> ISP operators will be wondering what that heck that spike was.
>
> If your phone is configured with Spanish, it will display the alert in
> both English and Spanish.
>
> “THIS IS A TEST of the National Wireless Emergency Alert System. No action is
> needed.”
>
> “ESTA ES UNA PRUEBA del Sistema Nacional de Alerta de Emergencia. No se
> necesita acción.”
>
> You'll know your iOS17 device did an extra data query, if it displays a
> longer message (extra sentences) in addition to the messages above.
>
> "This is only a test. No action is required by the public."
>
>
> https://www.fema.gov/press-release/20230803/fema-and-fcc-plan-nationwide-emergency-alert-test-oct-4-2023

Re: Test Lab Best Practices

[Aaron Gould]

I agree with others here...

Physical lab - gotta have console server for the most control - perle 
console server is good, and also good ole fashion cisco terminal server 
(2509/2511 or 2600 with asynch module)


Virtual labs are great for testing features and functionality

- Juniper vLabs

- Cisco DevNet sandbox

- Cisco CML (i think fka VIRL)

- EVE-NG

- GNS3

I use these virtual environments a lot and do videos about them on my 
youtube channel, where I try to cover some SP-related topics.  Hope it helps


https://jlabs.juniper.net/vlabs/

https://developer.cisco.com/site/sandbox/

https://www.youtube.com/@aarontechtalk

https://www.youtube.com/playlist?list=PL2ZMKm7ZEEWI8YyRWm9fnYNtRaV-fi-7x

https://www.youtube.com/playlist?list=PL2ZMKm7ZEEWLMVxuZqeXzciRu59C02NAc


-Aaron


On 9/28/2023 9:14 AM, Kenneth Vedder wrote:

Hello NANOG,

We have been struggling with firmware bugs from a specific router 
vendor. I am looking to set up a test lab of our core network and a 
few remote site routers.  Protocols would include SR-MPLS, ISIS, EVPN 
MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for 
any tips or resources anyone has that might cover testing strategies 
and/or best practices.


Thanks,
Ken


--
-Aaron

Re: Comcast contact sought

[Aaron de Bruyn via NANOG]

We get around the brain-damage by having our router grab all DNS requests and 
convert them to DoT or DoH using dnsdist. That probably won't work if you're 
hosting a DNS server on your cable connection though.

Call the normal support number and have them disable the "Security Edge" 
service. The "best" they can apparently offer is that it'll stay disabled until 
your modem gets a firmware upgrade or is factory reset. Then you'll have to 
call back in and disable it again.

Just be prepared that they're going to tell you it'll cost more for providing 
less service. Security Edge is horrible? Disabling it costs more. Don't need a 
phone number so Comcast can pad their numbers to the FCC? It'll cost you more. 
Same with not needing cable TV for your business. It costs you more because 
Comcast can't use you as a bargaining chip when negotiating with other media 
companies.

-A

On Sun Sep 24, 2023, 05:05 AM GMT, Al Whaley  
wrote:
> I am looking for a senior contact at Comcast.
>
> I have been trying to assist someone with a business connection that runs a 
> server farm. Recently the business cable modem started to short-stop port 53 
> for UDP and TCP. Apparently, a transparent DNS proxy somehow got activated 
> and all outbound traffic to any IPv4 or IPv6 address is intercepted and 
> handled by the modem – or not handled. Sadly, the proxy is stupid and a) 
> ignores the intended destination address, and b) drops things it doesn’t know 
> about, including any AXFR / IXFR and other more esoteric traffic, normal for 
> DNS server installations, but not used by the public. The DNS servers are not 
> able to do work, e.g. act as secondaries.
>
> I know others in the same configuration with servers that have been lucky and 
> not had this ‘feature’ activated, but I have found several references on 
> forums where people have been caught by this and unsuccessful in reaching 
> anyone in management, so it is a known problem.
>
> Comcast doesn’t allow customer supplied DOCSIS modems with multiple fixed 
> IPs. Other avenues exhausted as well.
>
> I’m hoping someone at Comcast can disable this. Attempts to go through 
> customer service… well we all know where that ends up. Escalations just don’t 
> go to anyone technical or interested.
>
> regards
> Al Whaley
> Sunnyside Computing, Inc.

Re: So what do you think about the scuttlebutt of Musk interfering in Ukraine?

[Aaron de Bruyn via NANOG]

> Starlink isn't a monopoly. Ukraine could have guided their munitions with 
> Iridium or another satellite Internet system.

Don't forget GLONASS. 

On Thu Sep 14, 2023, 03:10 AM GMT, William Herrin  wrote:
> On Wed, Sep 13, 2023 at 5:47 PM Michael Thomas  wrote:
>> Doesn't this bump up against common carrier protections?
>
> Hi Michael,
>
> Internet providers aren't common carriers. If they were, it'd be
> unlawful to stop your customers from sending email spam that was
> merely offensive rather than illegal. It's also why Internet providers
> aren't required to follow network neutrality. Internet providers gain
> their immunity through section 230 and the DMCA instead.
>
> Common carrier status typically applies to shipping companies and
> basic telephone service. Part of the mess with unwanted phone calls is
> that the caller has to break the law (e.g. by calling a number on the
> do-not-call list) before the phone company is allowed to act against
> them.
>> I sure don't
>> want my utilities weaponizing their monopoly status to the whims of any
>> random narcissist billionaire.
>
> Starlink isn't a monopoly. Ukraine could have guided their munitions
> with Iridium or another satellite Internet system.
>
> That said, volunteering services to the military of a nation at war
> and then pulling the rug out from under them is so classless, one
> wonders if Musk isn't trying to build a communist utopia.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: it's mailman time again

[Aaron de Bruyn via NANOG]

I donno Rich...a couple of decades ago I lost my Slashdot account because 
someone was able to access it.
I used the password in two places...Slashdot and all the blasted mailman 
instances I was signed up with.

To this day, I still use the same password on all my mailman subscriptions 
because I consider mailman insecure for emailing out passwords. I just 
obviously don't use the password anywhere else. So you're right that all anyone 
can do is unsubscribe me from something...which isn't a big deal, but it makes 
me wonder just how many people have terrible mailman passwords and maybe use 
them elsewhere...and wouldn't report a compromise because...well...it'd make me 
look stupid. 

Ignoring all of that—it's just a horrible practice to not encrypt passwords and 
to email them out. You don't really even need a mailman password. You just put 
in your email address and hit 'unsubscribe'...and it'll send you a link to 
click as authorization...so why not drop passwords altogether and just reply on 
click-to-authorize? Or just encrypt the passwords and have a "forgot password" 
click-to-reset like every other app on the planet?

-A

On Sat Sep 2, 2023, 07:57 AM GMT, Rich Kulawiec  wrote:
> On Fri, Sep 01, 2023 at 10:16:05AM -0700, Randy Bush wrote:
>> and i just have to wonder about sending passords over the net in
>> cleartext in 2023. really?
>
> This is a non-issue.
>
> Given that pretty much every SMTP connection is encrypted and that
> the worst thing that an attacker in possession of one of your Mailman
> passwords can do is unsubscribe you (in which case you and the list
> manager will be notified, and you can solve the problem quite rapidly),
> no, this isn't a problem that anyone needs to worry about.
>
> I've run (and am running) a lot of mailing lists with Mailman including
> some large-ish ones for what's now approaching 20 years. The scenario
> above has never happened. Nobody's even tried, which isn't surprising
> given that such an attack is increasingly difficult and yields little,
> if any, benefit to the attacker. Moreover, any hypothetical attacker
> possessing the resources and expertise required to pull this off could
> certainly find far more effective things to do.
>
> ---rsk
>

Re: MX204 Virtual Chassis Setup

[Aaron Gould]

some of these port capabilities are weird to me.  like on the 
ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?!


me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400
  48 0   1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 
4x10G 3x100G
  49 0   1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 
4x10G 3x100G
  50 0   1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 
4x10G 3x100G
  51 0   1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 
4x10G 3x100G
  52 0   1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 
4x10G 3x100G
  53 0   1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G 
4x10G 3x100G

  54 NA  1x10G




On 8/23/2023 11:29 AM, t...@pelican.org wrote:

On Wednesday, 23 August, 2023 16:33, "Mark Tinka"  said:

[faceplate oversubscription]


On the new ACX line, yes.

Not Trio, and different PLM :)


We don't mess around with any other MX products, so not sure (although
we are still yet to deploy the MPC10E's and the MX304).

MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the 
magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions 
beyond "SUM(port-speeds) <= 1.6T".

They make sense once you've looked at the block diagram for the thing and followed the 
lines, but things like "4x10G breakout can only go in odd-numbered ports, and you 
have to leave the corresponding next-lowest even-numbered port empty" are not 
instantly obvious.

Thanks,
Tim.



--
-Aaron

Re: Your input sought on PeeringDB's Network Type field

[Aaron Wendel]

I just left a couple sections blank.


On 6/14/2023 3:31 PM, Justin Streiner wrote:

Leo:

The survey might also want to include response options along the lines 
of: "Don't know / N/A".


Thank you
jms


On Wed, Jun 14, 2023 at 12:18 PM Leo Vegoda  wrote:

Hi,

PeeringDB's Product Committee wants your input on whether the Network
Type field is useful. Should it go? Should it change?

We have published a very short blog post describing the options and
linking to the survey.

https://docs.peeringdb.com/blog/network_type_your_input_sought/

Your input will influence our decision.

Thanks,

Leo Vegoda for PeeringDB's Product Committee



--
====
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: Comcast Business Account Website Broken

[Aaron de Bruyn via NANOG]

Someone else here gave me a pointer when I was running into this on the USPS 
site.

Clear your cookies for that site. (In Chrome/Edge, go to the site, open up the 
dev tools, go to the "Application" tab, find cookies, delete them all).

Something probably went a little nuts with the site and ended up creating too 
much data in one or more cookies.

-A

On Wed Jun 14, 2023, 06:20 PM GMT, Matt Hoppes 
 wrote:
> For the last two weeks we have been unable to pay any bills on the
> business Comcast website.
>
> Clicking on any billing link results in:
>
> 400 Bad Request
> Request Header Or Cookie Too Large
>
> This is going to the URL of:
> https://business.comcast.com/oauth/oauth2/authorize?client_id=comcast-business-myaccount-prod_type=code_uri=https%3A%2F%2Fbusiness.comcast.com%2Faccount%2Fbilling
>
>
> Even trying to log out from the customer portal results in:
>
> Sorry
>
> Something went wrong. Please check back later.
>
> This is going to this URL:
> https://business.comcast.com/account/logout
>
> Hoping someone on here can get this to the right people to fix. I'm
> sure Comcast would love to get payments from their commercial customer base.

Re: Issues with prefix / help needed

[Aaron Gould]

yeah i see what you mean by, it doesn't work, then it starts working...


i traced to it, and it wasn't responding at first, then later it worked


C:\>tracert -w 1 86.104.228.1

Tracing route to 86.104.228.1 over a maximum of 30 hops

...

  9   118 ms *  119 ms prs-bb1-link.ip.twelve99.net 
[62.115.112.243]
 10   125 ms   124 ms   126 ms  ffm-bb1-link.ip.twelve99.net 
[62.115.123.12]

 11 *    *    * Request timed out.
 12 *    *    * Request timed out.
 13   133 ms   133 ms   133 ms 
ipmax-ic340750-zch-b2.ip.twelve99-cust.net [62.115.168.201]

 14   130 ms *  130 ms  po5.er01.zrh56.ch.ip-max.net [46.20.254.13]
 15   128 ms   129 ms   129 ms three-fourteen.cust.zrh56.ch.ip-max.net 
[46.20.240.71]

 16 *    *    * Request timed out.
 17 *    *    * Request timed out.
 18 *    *    * Request timed out.
 19 *    *    * Request timed out.
 20 *    *    * Request timed out.
 21 *    *    * Request timed out.
 22 *    *    * Request timed out.
 23 *    *    * Request timed out.
 24 *    *    * Request timed out.
 25 *    *    * Request timed out.
 26 *    *    * Request timed out.
 27 *    *    * Request timed out.
 28 *    *    * Request timed out.
 29 *    *    * Request timed out.
 30 *    *    * Request timed out.

Trace complete.

C:\>tracert -w 1 86.104.228.1

Tracing route to 86.104.228.1 over a maximum of 30 hops

...

  9   119 ms   118 ms   118 ms prs-bb1-link.ip.twelve99.net 
[62.115.112.243]
 10 *  125 ms   124 ms  ffm-bb1-link.ip.twelve99.net 
[62.115.123.12]

 11 *    *    * Request timed out.
 12 *    *    * Request timed out.
 13   132 ms   132 ms   133 ms 
ipmax-ic340750-zch-b2.ip.twelve99-cust.net [62.115.168.201]

 14   129 ms *  129 ms  po5.er01.zrh56.ch.ip-max.net [46.20.254.13]
 15   129 ms   129 ms   129 ms three-fourteen.cust.zrh56.ch.ip-max.net 
[46.20.240.71]

 16   129 ms *  129 ms  86.104.228.1

Trace complete.

C:\>





On 3/25/2023 3:54 AM, ic wrote:

Hi there,

I’m contacting you because after spending 2 days troubleshooting I can’t seem 
to find a solution to the following.

We (AS45021) bought/transffered the 86.104.228.0/24 prefix a few months back 
because we couldn’t wait longer on the RIPE waiting list.

Before you ask, yes, AS45021 is currently single homed, this will change in a 
week (it requires travelling a few hundred miles and I couldn’t do it before).

Since we started announcing this prefix, things have been spotty, at best. 
While it seems visible in all the looking glasses I tried, it spends sometimes 
hours, sometimes days, being unreachable (you can try for ex. 86.104.228.1 or 
86.104.228.26).

I have full access (up to packet capture) on the AS and its upstream. When I 
ping one of the IPs from various ISPs, I see the ICMP Echo Request and Reply on 
the wire, going where it’s supposed to go, but it doesn’t reach the pinging 
host. Pinging any IP of the upstream (AS42275 / 85.208.69.0/24 in this 
location) works.

ROAs and RPKI seem fine to me.

I’m starting to suspect that maybe the previous user of the prefix is still 
announcing it somewhere and “shouting louder” than me. It seems when I clear 
sessions, it immediately works for a while, then stops.

Do you all have any idea what I should check / try next?

BR, Michel


--
-Aaron

Re: Spamhaus flags any IP announced by our ASN as a criminal network

[Aaron Wendel]

The solution to your problem is to terminate the customer causing the 
abuse, in this case 62yun.com.  Once you do that I'm sure Spamhaus will 
stop listing all your IPs.


Aaron


On 3/20/2023 6:54 AM, Brandon Zhi wrote:



It seems you've reached the point that they ignore specific
prefixes and set every prefix you are advertising as criminal.

*
*
Our sponsor (LIR) 62yun.com <http://62yun.com>, they have 2 prefixes 
for VPS/Dedicated Server using our ASN.*

*
62yun did receive a lot of complaints, but as far as I know they have 
been handling them (their head said their team is not good at English 
and so they did not reply emails)
For me, I cannot reply to all emails for them, since I don't have that 
much time. I also need to work for my company.



As I understand it, most things at Spamhaus are manual determinations.
You click on "show details" and they give you a list of timestamped
report IDs, each with a 1-line description of the reviewer's
assessment of the fault.


I checked https://check.spamhaus.org/listed/?searchterm=46.23.100.0 
and the reason they gave us was simple, saying our not willing to 
handle abuse. but we stressed with them many times that we are 2 
different companies. We also do not have the authority to handle these 
complaints, but we will alert 62yun.com <http://62yun.com>.


But they still intend to blacklist all the prefixes under our ORG ID, 
even if the user is not us.



Based on my past experiences, Spamhaus is rather gracious at
first, but if you ignore them, they will start blocking you en
masse. About 10 years ago, I worked for a datacenter/NSP and
personally handled all Spamhaus complaints, and as soon as I left
to go to another company (and the company stopped taking care of
the complaints), Spamhaus blocked every single one of their IPs
until they committed to actually handling the complaints again.



This has little impact on 62yun.com <http://62yun.com>'s VPS business, 
and my feeling is that if someone uses their VPS to build a mail 
server those emails that are sent from this server may be rejected.


However, we are recently building a CDN for one of our partners (a 
social media company), and we need to use a provider like vultr, which 
is not really an IP Transit provider, to announce prefixes, however, 
they reject prefixes on the Spamhaus list.


I don't think any ISP would reject an IP that is on the Spamhaus list.


*Brandon Zhi*
HUIZE LTD

www.huize.asia <https://huize.asia/>| www.ixp.su 
<https://www.ixp.su/> | Twitter



This e-mail and any attachments or any reproduction of this e-mail in 
whatever manner are confidential and for the use of the addressee(s) 
only. HUIZE LTD can’t take any liability and guarantee of the text of 
the email message and virus.




On Mon, 20 Mar 2023 at 02:29, Tim Burke  wrote:

Have you received complaints from Spamhaus in the past? If so,
have you acted on them in a timely manner?

Based on my past experiences, Spamhaus is rather gracious at
first, but if you ignore them, they will start blocking you en
masse. About 10 years ago, I worked for a datacenter/NSP and
personally handled all Spamhaus complaints, and as soon as I left
to go to another company (and the company stopped taking care of
the complaints), Spamhaus blocked every single one of their IPs
until they committed to actually handling the complaints again.

V/r
Tim



On Mar 18, 2023, at 8:57 AM, Brandon Zhi  wrote:

Hello guy,

We recently discovered that any IP address announced by our ASN
is blacklisted by Spamhaus, even if we only announced it but not
use it.

I would like to ask if this is manually set by Spamhaus or is the
system misjudgment? Has anyone encountered the same situation as us?


Best,

*Brandon Zhi*
HUIZE LTD

www.huize.asia <https://huize.asia/>| www.ixp.su
<https://www.ixp.su/> | Twitter


This e-mail and any attachments or any reproduction of this
e-mail in whatever manner are confidential and for the use of the
addressee(s) only. HUIZE LTD can’t take any liability and
guarantee of the text of the email message and virus.





--
====
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Aaron de Bruyn via NANOG]

If we have downtime, we lose revenue, customers, sleep, etc...

If the government does it, what are you going to do? Get your license somewhere 
else?

-A

On Sat Feb 25, 2023, 11:39 PM GMT, Christopher Morrow 
 wrote:
> On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:
>>
>> Verizon network maintenance will impact access to the “National Driver
>> Register,” a system that motor vehicle offices around the country need to
>> check before handing out a license.
>
> Wait, what year is it?
> how is a network maintenance on what seems like a fairly critical system going
> to cause a total outage of said system?
>
> I think we time traveled back to 1990 here...
>>
>> All 50 states and D.C. participate in the National Driver Register, a
>> database maintained by the National Highway Traffic Safety Administration.
>> The register contains information about drivers who have had their driving
>> privileges revoked, suspended or denied due to serious traffic violations,
>> such as driving under the influence of alcohol or drugs, reckless driving
>> or excessive speeding.
>>
>>
>> The scheduled maintenance should be finished by Monday, in case you needed
>> to update your driver's license or planned to do some reckless driving
>> this weekend.

MX204 and MPC7E-MRATE EoL - REVOKED

[Aaron Gould]

Did you hear? EoL was revoked December 2022... I'm so glad, I like and 
use the MX204 and the MPC7E-MRATE



TSB69626 - 12/5/2022 - Revoke End of Life Announcement: MX204

https://supportportal.juniper.net/s/article/Revoke-End-of-Life-Announcement-MX204 
<https://supportportal.juniper.net/s/article/Revoke-End-of-Life-Announcement-MX204>



TSB69631 - 12/2/2022 - Revoke End of Life Announcement: MPC7E-MRATE, 
MPC7E-MRATE-RTU


https://supportportal.juniper.net/s/article/Revoke-End-of-Life-Announcement-MPC7E-MRATE-MPC7E-MRATE-RTU 
<https://supportportal.juniper.net/s/article/Revoke-End-of-Life-Announcement-MPC7E-MRATE-MPC7E-MRATE-RTU>


-Aaron

Re: txt.att.net outage?

[Aaron de Bruyn via NANOG]

txt.att.net  is returning MX records and those machines 
don't have port 444 open...

Wouldn't you want to be sending something like a SNPP message instead? It's a 
much less convoluted delivery process and is almost real-time (no queuing).

I guess it's been a decade or so since I've dealt with emergency services and 
paging...is SNPP even a thing anymore?

I looked at some old code I wrote 
(https://github.com/darkpixel/snppsend/blob/master/more-providers 
), and it 
doesn't look like snpp.attws.net  exists.

-A

On Fri Jan 20, 2023, 02:12 PM GMT, William Herrin  wrote:
> On Thu, Jan 19, 2023 at 8:09 PM Dan Walters via NANOG  wrote:
>> Know this is a longshot, any chance anyone from the txt.att.net domain might 
>> be able to help us with what we believe is a blacklist block or possibly an 
>> outage?
>> We deal with 911 cad dispatching and is affecting first responders so 
>> looking to see if there is a faster way to resolution.
>
> Hi Dan,
>
> As I understand it, txt.att.net is a low-volume courtesy service not
> intended for important communications. A paid service like Twilio can
> handle production-grade SMS delivery.
>
> Regards,
> Bill Herrin
>
> --
> For hire. https://bill.herrin.us/resume/

AS3491 Contact

[Aaron Atac via NANOG]

Hi,

Would someone from AS3491 please contact me off-list?

Been trying to fix a prefix acceptance issue for weeks and am getting nowhere.

Thanks,
Aaron

Re: [External] Normal ARIN registration service fees for LRSA entrants after 31 Dec 2023

[Aaron Wendel]

I'm not trying to troll, this is a serious question:

Is there a formal agreement that says that all legacy resources will 
receive free registry services forever and ever or is it just an 
informal "That's how it was done"?


Aaron

Re: Looking for contact within Comcast Xfinity

[Aaron C. de Bruyn via NANOG]

I ran into this a few days ago.

Both the random agent I talked to and our sales rep said they can't disable
the security edge service without increasing the cost of service for all of
our accounts.

Apparently it costs more to not molest DNS traffic leaving your network.

They can temporarily disable it, but they said it will turn back on when
the modem is rebooted.

It seems to only affect TCP and UDP port 53.

I fixed it by setting all of our routers to use DoH and DoT exclusively.
They can't intercept and molest that traffic.

-A



On Tue, Aug 23, 2022, 05:39 Michael Brown  wrote:

> If anyone from Comcast Xfinity is on this list, can you please reach out
> to me?
>
> We're getting increased reports of xFi Advanced Security customers being
> unable to access hosted sites and attempting to open tickets has had no
> success.
>
> Thanks,
>
> Michael Brown
>
>

Re: 2 Byte ASNs??

[Aaron Wendel]

We've never had an issue requesting a 2-byte ASN from ARIN.  Our last 
request was, maybe, a month ago.


Aaron


On 8/5/2022 10:16 AM, Justin Wilson (Lists) wrote:
Whats the availability of two byte asns look like? Anyone able to 
obtain one recently? I have a network that is all Mikrotik and the 
route targets are messing with them.  They can’t use communities with 
their 4 bytes asn.  It’s one of those it really isn’t a big deal but I 
thought I would ask.




Justin Wilson
j...@mtin.net

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

AS 10797 Charter Communications Contact

[Aaron M. Pace]

If anyone from Charter could contact me off list would appreciate it. 
Troubleshooting an issue for slow performance to remote users using with a 
hardware IPsec appliance. If I use a different egress circuit from my network, 
fixes the issue. It seems to be resolved after a few days, and the transit AS 
(Lumen) does not see any issues in their path. 

Thanks
Aaron

Re: Reporting Comcast outside plant issues?

[Aaron de Bruyn via NANOG]

I had that during the 2020 storm that swept through the US. I called PUD a few 
months before about a tree hanging at a 45 degree angle above the primaries. I 
called again a month later when I noticed the tree had been slowly shifting. No 
sense or urgency from the PUD. Then the storm hit and I watched from my car as 
it smashed into a pole, snapped the primaries, destroyed a transformer, snapped 
the secondaries, snapped the pole, and then hung bits of itself from the cable 
space. It was pretty spectacular—I wish I had gotten it on video.

Ignoring it for ~2 months turned a $500 tree removal into something that cost 
tens of thousands of dollars—not to mention the teams that had to do all the 
work in sub-freezing temps instead of cool with intermittent showers.

Everyone on in a ~1 mile stretch went without power for ~17 hours in 13 degree 
weather. Fortunately I have two generators that are worth more than my car and 
I had the ability to fail over to a Starlink connection. Internet was back up 
about about 38 hours later.

-A

On Mon Jun 27, 2022, 05:14 PM GMT, Mike Hammett  wrote:
> Maybe.
>
> I saw multiple reports of a town this past week end that didn't respond to 
> multiple calls for a transformer and pole CURRENTLY on fire. I guess they had 
> better things to do.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
[image] [image] 
[image] 
[image] 

> Midwest Internet Exchange 
[image] [image] 
[image] 

> The Brothers WISP 
[image] [image] 

--
> From: "Jay Hennigan" 
> To: nanog@nanog.org
> Sent: Monday, June 27, 2022 12:07:16 PM
> Subject: Re: Reporting Comcast outside plant issues?
>
> On 6/26/22 19:27, Justin Streiner wrote:
> > Does anyone here have a contact at Comcast for reporting outside plant
> > issues that are not (at the moment) service-affecting? I am not a
> > Comcast customer, and they make it nearly impossible for non-customers
> > to reach them unless you're signing up for service.
>
> Call the non-emergency number for your local PSAP (police or fire
> department) and report wires down. They'll know how to get it handled.
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>

Comcast: "Reloading Statics" today

[Aaron de Bruyn via NANOG]

Just a heads-up for the Comcast crew lurking here...

I've had 3 different cable connections (Oregon and Washington State) go
down in the last ~1.5 hours.
Staff on-site have tried rebooting the modem with no success.
When we call support, they say something along the lines of "huh, that's
odd...your modem is online, but the statics aren't loaded".
They then proceed to reload the statics and everything comes back up.

The third time I commented to the support rep that this was my third call
today and she replied that she had taken "several" calls about the same
issue from other customers.

Not sure if there's a new firmware update going our or what, but the issue
appears to be on the Comcast end.

-A

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Aaron Wendel]

On 5/24/2022 10:48 AM, Mitchell Tanenbaum via NANOG wrote:


I have two fixed wireless Internet connections here.  One is 25/5, the 
other is 35/5.  There is no cable, no fiber, no cellular, not even DSL 
from the phone company.  That is reality in metro Denver, CO 
(actually, the foothills, 25 miles from the state Capitol building).


Regarding Starlink, no, you can’t get it.  I paid my deposit a year 
and a half ago and I am still on the waiting list.  Every time that I 
get close to the date they promise, they change the promise. Maybe I 
will get Starlink service some time in the future, but, not any time soon.


Oh, yeah, and 25 meg down costs $75 a month.  If you want VoIP, that 
is another $20+.


So not only is it slow, it is expensive too.

So yes, there still is a problem, right here in America.  And not just 
in the boonies.


Mitch



This brings up another issues no one is really talking about and that's 
affordability.  We're about to lower our price on 10G to the home to 
$50/mo because that was the number the FCC would pay people who 
qualified.  Now they've lowered that subsidy to $30.  The pandemic 
exposed the fact that there are a lot of people out there that just 
can't afford the current pricing structure.  We give a gig away for free 
with a one time install fee and we had people calling us who's kids were 
at home for school and they couldn't afford the $25/mo we'd break their 
$300 install into.  We ended up just waiving a ton of fees during those 
early COVID days.


Aaron

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Aaron Wendel]

On 5/24/2022 9:57 AM, Forrest Christian (List Account) wrote:
If the government is going to fund anything at all anymore, it needs 
to be fiber all the way to the home which is built and managed in a 
way that any provider can use it.   This probably means a single 
strand from each home to some concentration point no more than 10km 
from the home and then a backbone/middle mile supporting several 
carriers from that point.   The position of this concentration point 
to be determined by the density in the area.


In an ideal world, yes, this is exactly how it would work although there 
would be some logistical issues.


If you sit in these hearings the various government entities hold and 
listen to Charter's "Government Affairs Representative" then that is 
absolutely not true and coax is the wave of the future.

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Aaron Wendel]

The Fiber Broadband Association estimates that the average US household 
will need more than a gig within 5 years.  Why not just jump it to a gig 
or more?



On 5/23/2022 1:40 PM, Sean Donelan wrote:


https://www.fcc.gov/document/fcc-proposes-higher-speed-goals-small-rural-broadband-providers-0 



The Federal Communications Commission voted [May 19, 2022] to seek 
comment on a proposal to provide additional universal service support 
to certain rural carriers in exchange for increasing deployment to 
more locations at higher speeds. The proposal would make changes to 
the Alternative Connect America Cost Model (A-CAM) program, with the 
goal of achieving widespread deployment of faster 100/20 Mbps 
broadband service throughout the rural areas served by rural carriers 
currently receiving A-CAM support.

Re: Cogent ...

[Aaron Wendel]

I've used Cogent for years and have never been asked to sign an NDA with 
them.


Of the 4 providers I use regularly they are the second highest price so 
I wouldn't consider them cheap any more either.


There's no better or worse than any transit provider these days.

Aaron

On 3/31/2022 10:38 AM, Laura Smith via NANOG wrote:

Hmmm

Spring has sprung and the waft of drivel from a new season Cogent salesdroid 
filled my telephone earpiece today.

I've never liked the Cogent way of business and my understanding of their IP transit is 
that it falls into the "cheap for a reason" category.

However, perhaps someone would care to elaborate (either on or off-list) what 
the deal is with the requirement to sign NDAs with Cogent before they'll 
discuss things like why they still charge for BGP, or indeed any other 
technical or pricing matters. Seems weird ?!?

Laura


--
====
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

Thanks Jason—they are all are business connections. I know they can be
restarted through the business portal, but honestly the business portal is
terrible for large clients.
Not all our connections are listed under the same account due to something
with Comcast and the way "regions" and various services are split. i.e.
there doesn't appear to be any way of seeing EDI circuts, only cable
connections. We were basically told for all the cable connections that we
had to have multiple email addresses for the various groups (i.e.
i...@example.tld could add accounts in one region, i...@example.tld could
handle a different region, etc...)

In some cases a reboot will trigger a pull of the latest firmware, which
might include security fixes, performance improvements, and other changes.

Good to know...but there generally seems to be a culture in both the "home"
side and the premier side that Comcast modems *will* start having problems
if they are left up and running for more than ~1 month. Some people say
they need to be rebooted every 21 days, others say "three to four weeks",
etc...

My routers run FreeBSD and they get rebooted maybe once a year when there's
a security update that affects us. Granted, the Comcast modems are probably
some embedded Linux variant, but it seems odd to me that there's this
generally accepted idea that they just need to be rebooted every few weeks
to "clear stuff out". They aren't my grandmother's Windows PC.

High packet loss typically suggests an RF impairment of some type. I don’t
know how to explain the PoE comment but am happy to look at your connection
if you want to email me off-list.

I would suspect RF issues on the cable side as well. There's a very large
cell phone tower about 2 blocks away with about half a million antennas on
it.

As for the PoE issue, I'm not trying to get anyone canned. The tech was
professional and polite—just wrong. He was insistent enough that I started
to question my understanding of the PoE standard, my networking knowledge
and possibly my own sanity. 

Thanks for reaching out though. I really do appreciate how responsive
Comcast is—not just you and others on NANOG, but generally as a large
corporation as a whole. For ~40 connections I usually find myself reaching
out to the premier group once a week and they're polite, knowledgeable and
99% of the time 'hassle free'. 

-A

On Wed Mar 30, 2022, 05:53 PM GMT, Livingood, Jason
 wrote:

> I asked him to remotely reboot the modem because there was high packet
loss.



FWIW, as a customer (assuming residential), you can login to the website
and check for area outages/impairments at
https://www.xfinity.com/support/status-map. You can also use the Xfinity
app to remotely reboot your cable modem, run diagnostics/check for outages,
etc. See https://www.xfinity.com/support/articles/check-service-outage



> Both times I've talked with him, he noted the high packet loss, started
to reboot the modem, and then asked me point-blank if we had any PoE
switches on our network.



High packet loss typically suggests an RF impairment of some type. I don’t
know how to explain the PoE comment but am happy to look at your connection
if you want to email me off-list.



> I said "it's up and working fine, why would I reboot it?".



In some cases a reboot will trigger a pull of the latest firmware, which
might include security fixes, performance improvements, and other changes.



Jason

Re: PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

Thanks Blake,

As I understand it all that stuff is on the "cable provider" side of the
CPE and (within reason) it's up to the provider to deal with the signals
arriving on the cable side of the modem.
i.e. if it was a blower or something in our suite that was causing RF
interference, the provider might work with us to move the modem or the
cable run.

-A

On Tue Mar 29, 2022, 09:59 PM GMT, Blake Hudson  wrote:


On 3/29/2022 3:24 PM, Joe Greco wrote:

He's got graphs showing it every 24 hours? Liar, liar, pants on fire,
lazy SOB is looking for an excuse to clear you off the line. Where the
heck does this "24 hour" cycle even come from? What SNMP OID is there
for "ghostly PoE build-up"? What crontab is there that would clear out
such buildups in the router's daily run? What capacitor would store up
juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS
WANT TO KNOW!


Taken at face value, I assume the tech would be looking at historical
signal graphs (we keep them for cable networks for each CM) that record
stats like FEC, SNR, and signal strength. For aerial runs it's common to
see some change throughout the day due to warming and cooling. These
look like waves with peaks and valleys around 4PM/4AM and generally
affect all customers in a service area equally. Sometimes there will be
a device at a customer premise that causes interference with a CM,
something like a motor or tool. These could absolutely be on a 24hr
cycle (think of a programmable thermostat kicking on the blower fan in
your HVAC at the same time every day).

As Joe said, there's no SNMP MIB for PoE buildup. There are well
documented MIBs for DOCSIS to cover standard signal level, quality, or
similar. The cause of that signal strength or quality can be myriad.
This Comcast tech has likely climbed the ladder of inference several
steps too far.

Re: PoE, Comcast Modems, and Service Outages

[Aaron C. de Bruyn via NANOG]

On Tue, Mar 29, 2022 at 1:12 PM Joe Greco  wrote:

> So if you want the $100 test to eliminate PoE electrical effects, get
> a pair of media converters and run fiber between them.  Put the CPE on
> the far end.  Optimize as appropriate if you have SFP-capable switches.


Sure--that would shoot down the "leaking non-existent PoE across a
motherboard and out another NIC" theory, but I was more thinking along the
lines of something like PoE causing RF interference or something.
I mean it's DC not AC so...it wouldn't be putting out a modulating signal
that interferes? ...honestly that's outside my knowledge domain.

-A

Re: PoE, Comcast Modems, and Service Outages

[Aaron C. de Bruyn via NANOG]

On Tue, Mar 29, 2022 at 12:20 PM Brie  wrote:

> Unifi/EdgeSwitch?
>

Yeah.  Unfortunately.  USW-24-250.


> Yeah, you know when 24v passive POE is turned on because it kills the
> port on the other end that aren't designed to handle it.  Your router
> would likely have a dead eth port on it.
>

I've never tested it with one of my routers, but a tech did accidentally
test it on a UniFi AP.  I'm still not sure how he ignored the warning about
sending 24 volts down the line, but the WAP didn't like it and decided to
refuse to work with anyone ever again. ;)


-A

Re: PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

Just to be clear Josh, I'm not insulting him.

I find the situation extremely difficult to believe based on my (possibly
incorrect) understanding of how PoE works and very (very!) basic knowledge
of things like RF interference—especially when it comes to Cable networks.

I mean, the call literally went like this:
"Thank you for calling Comcast this is , how can I help you?"
"Hey, can you remotely reboot the modem on account 12345? We're seeing high
packet loss and latency starting about 10 minutes ago."
"Yeah...uh...do you have a PoE switch at that location?"

When you hear hoof beats, look for horses, not zebras. As a first
troubleshooting step, I certainly wouldn't jump to "it's PoE". Granted, I
have no idea if Comcast has "PoE Buildup" graphs in their internal tools,
but based on my conversations with tons of other Comcast reps about tons of
other Comcast connections and never hearing one of them mention those
graphs, I'm leaning towards him lying through his teeth.

Lastly, the reboot of the Comcast modem "fixed" the issue.

I saw one of the IT guys from another office in the complex a few minutes
ago and he said their internet had problems at the same time. Comcast has
been out to the equipment room in the facility ~5 times over the last few
years to "adjust" things...so I'm still leaning towards this being
something more common like faulty equipment, bad signal levels, etc...and
not "It's because you have a PoE switch".

-A

On Tue Mar 29, 2022, 07:42 PM GMT, Josh Luthman
 wrote:

There's a certain manufacturer of TDD radio where the CPU clock is at the
same frequency as what Verizon's enodeB will transmit.  Even at miles away,
it can and will cause PIM issues.  Again, don't rule it out.

Maybe he's just looking for a simple answer that 99% of callers will accept
and it makes them happy.  When a customer of mine tells me they think it's
something and I know it's off, I just let them believe in their statement.
There's no reason to go after this tech and insult him, all that's doing is
making everyone miserable.

On Tue, Mar 29, 2022 at 3:26 PM Joe Greco  wrote:

> On Tue, Mar 29, 2022 at 03:07:47PM -0400, Josh Luthman wrote:
> > We've routinely seen where lines not even connected to the same circuit
> in
> > any way (ie an OTA antenna coax line and cat5 POE) cause issues with one
> > another.  As much as we would all love to have a perfect line in the
> sand,
> > there isn't.  Don't rule anything out until the issue is resolved.
> >
> > As someone that sees this in the field and watches people simply hate on
> > someone because there's a frustrating situation, it's worth taking a
> breath
> > before too upset.
>
> You can run cable lines next to A/C wiring and get problems too.  Or
> ethernet lines next to A/C wiring.  That does not justify wild claims
> about PoE such as what this tech was making, and until someone shows
> me a graph of "PoE buildups" observable via SNMP or whatever the
> cable company is using to graph trends, it seems pretty clear that
> this is a bogus answer.
>
> There's a lot of difference between "we observed this very specific kind
> of interference related to PoE in a particular circumstance" and the
> crazy generalizations being made by the tech.  Asking to please make sure
> your switch is grounded properly?  That'd be good.  Asking for PoE to be
> disabled on the port?  Yeah fine.  Suggesting separation of cables?
> Sure.  Checking for proper grounding of the ground block (on the cable
> inlet)?  Sure.  There's room for things to happen.
>
> I'm all for investigating with an open mind, but I draw the line at crazy.
>
> Given that so much of the world works on PoE, it seems like the other
> potential resolution would be to note that there's an implication here
> by the tech that Comcast's hardware is standards noncompliant and ask
> them what they plan to replace their cheap CPE with.
>
> ... JG
> --
> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
> "The strain of anti-intellectualism has been a constant thread winding its
> way
> through our political and cultural life, nurtured by the false notion that
> democracy means that 'my ignorance is just as good as your
> knowledge.'"-Asimov
>

PoE, Comcast Modems, and Service Outages

[Aaron de Bruyn via NANOG]

I just got off the phone with a Comcast tech, and wanted to double-check my
sanity.

Somehow in the last 6 months I've managed to reach the exact same rep twice
when dealing with an outage or a degraded service event.

I asked him to remotely reboot the modem because there was high packet loss.

Both times I've talked with him, he noted the high packet loss, started to
reboot the modem, and then asked me point-blank if we had any PoE switches
on our network.

When I said "yes", he said I needed to disable PoE because it messes with
the Comcast modems and he can see "buildups" in his graphs that show power
is "leaking" to the Comcast modem every 24 hours.

For reference, our setup is:

Internal Network ←→ PoE Switch ←→ My Router (FreeBSD Box) ←→ Comcast Modem

I told him the Comcast modem isn't plugged into the PoE Switch, it's
plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+
and the switch shows PoE isn't being send to My Router's LAN port. While
the switch is capable of outputting old-school 24v PoE, it must be
specifically turned on for a port, and it's not enabled or used anywhere on
the networks I manage.

When provided with that information, the Comcast tech still insisted that
the switch was sending PoE to My Router and it was "leaking through" to the
Comcast modem and that's why every 4-6 weeks the Comcast modem needs to be
reset. The tech insisted that switches that *are* PoE-capable *always* send
PoE even if the device doesn't request it or negotiate it. Attempts to
explain the difference between the old 24-volt PoE and PoE+/++ were met
with arguing that he's been in the industry for decades and I don't know
what I'm talking about...and that all my problems would go away if I just
disabled PoE everywhere on the switch.

Again, I double-checked the port and said "It's not sending PoE to my
router, but even if I were, I highly doubt PoE would leak through a PCI
card to the opposite side of the chassis to the on-board NIC and out to
your modem".

He insisted it happened "all the time" and he had previously fried
equipment by plugging it into a PoE switch. He insisted that he's also
handled quite a few calls relating to this magic PoE problem over the years
and Comcast has internal tools that show graphs of how much PoE power
"builds up" inside their modems and he "can see a buildup in my router that
resets every 24 hours".

I didn't have the heart to tell him that I manage about 40 networks that
have Comcast connections...and they *all* have identical FreeBSD boxes
acting as their router, and they are *all* using the exact same PoE
switches at every location with all ports set to PoE+...and we only have
degraded service or outages after ~30 days at ~3 locations.

Slightly off-topic, but if I call Comcast about outages or degraded service
and any *other* tech but this guy answers, they all say "you need to unplug
your Comcast modem and plug it back in once every 3-4 weeks" and they act
like it's normal to reboot the modems every few weeks. In fact, last week I
wanted Comcast to check on a modem setting at one location and they said
the modem had been up for over 127 days and it should be rebooted. I said
"it's up and working fine, why would I reboot it?".

Anyways, am I insane for thinking the tech was flat-out wrong? I
mean...occasionally some really bizarre stuff happens in IT...but this
seems extremely far-fetched and contrary to everything I know about the PoE
standard.

-A

Re: "Permanent" DST

[Aaron C. de Bruyn via NANOG]

On Tue, Mar 15, 2022 at 3:09 PM Joe Greco  wrote:

> We COULD all work in UTC and un-learn the weird system of hour offsets
> and timezones.  This would be convenient for people at a distance, since
> it would be simply a matter of stating availability hours, rather than
> giving someone hours AND a timezone and making them do the math.  If I
> say that I'm available for an hour at 22:00 UTC, that works out anywhere
> on the globe.  But do you know what timezone "CDT" is?  When's "17:00 CDT"?


Seems like an issue that could be solved by some simple tech that I'm
surprised Apple and Google haven't really implemented.

My sister is a "world traveler".  I have no idea what country she'll be in
next week.  If I decide to call her, I have no idea what timezone she'll be
in...let alone what "normal sleeping hours" are for her when she's
jet-lagged after a 14 hour flight.

I just call her phone and see if she answers.

I think just about every smartphone has a rudimentary "do not disturb"
feature built in.  My Google phone automatically switches to DND when it's
on the charging stand after 10 PM and turns off when I pick it up in the
morning.

The multitude of chat apps have presence.  Online, available, free to chat,
busy, unavailable, offline, do-not-disturb.

Why doesn't that exist for phone numbers? Create a public queryable server
that shows a status for a phone number.  Set your status to some
pre-defined value or make a custom status:
{
  status: "doing my taxes",
  do-not-disturb: true,
  emergencies: true,
  typical_availability: {
start: "14:00:00 GMT",
end: "04:00:00 GMT",
  }
}

I know FreePBX has presence support internally for extensions.  Come up
with a standard, integrate it with cell phones and you've solved
interrupting people because you don't know what arbitrary time numbers and
offsets they are using.

Android and iOS could have a 'master switch' on every phone.  Set your
status and all your various apps can pick up that status including voice
calls.
Android (and I'm sure iOS has it too) provides a way to say "these contacts
can override DND".

All that's left to solve is in-person stuff...which already currently sucks.

"My flight leaves at 6 AM local time and lasts 90 minutes, but I'm crossing
3 timezones heading west...so you need to pick me up at...uh4:30 AM
your time?  Oh waitare you currently in DST or not because we don't do
DST here, but I think you doso you either need to pick me up at 4:30 AM
or 3:30 AM...I'm not surewhat's your time is it now?  Ok, it's 5 AM my
time and 7 AM your time, so no DST, so...uh...but next week your zone is
switching to DST but we're already on it..."

vs

"My flight leaves at 06:00 zulu, lasts 90 minutes, so I'm landing at 7:30
zulu.  See you then."

For the record, I was always told DST was implemented because of farmers.
I'm a farmer and I hate timezones.  I just wake up when the rooster starts
crowing, and no one goes out to adjust him twice a year for DST.

-A

Re: Is soliciting money/rewards for 'responsible' security disclosures when none is stated a thing now?

[Aaron de Bruyn via NANOG]

I had a situation like that a few years ago.

Someone accidentally included the .git directory in a docker image that was
deployed to a customer's website.
Unfortunately early checkins of the .git directory included a copy of the
WordPress (yuck!) config file with hard-coded passwords. Those were moved
to environment variables, but never changed. And for some reason the
"developer" left indexing turned on. So the person was able to download the
git directory and walk back through the history and found the
passwordsand then connected to the database which had some mild PHI
(first names and phone numbers).

Since the tech contact for the domain came back to my company and not the
developer, they reached out to me. After a few pleasant emails back and
forth he told me exactly where he found the passwords. I rotated passwords
and yelled at the developer, and thanked the guy who found it. He kindly
asked if I would "donate" to him by buying something from his Amazon
wishlist. I should note that he asked *after* he told us exactly what the
problem was.

I discussed it with the client and they picked some ~$400 item from the
list and sent it to him.

It could have been worse, but everyone involved agreed that it would be
nice to reward the guy for pointing out the blunder.

$400 was a small price to pay for the client since they do something like
$10 million USD per month. After that the client paid for a full security
audit of their web presence by a 3rd party company and everything came back
clean.

Do what you think is appropriate, but I'm all for encouraging responsible
and positive disclosure as well as being kind. If the guy had started the
email with "send me money or else I'll disclose" the entire process would
have been very different.

-A

On Wed Mar 2, 2022, 10:30 PM GMT, Brie  wrote:

I just got this in my e-mail...

--
From: xxx 
Date: Thu, 3 Mar 2022 03:14:03 +0500
Message-ID: 
Subject: Found Security Vulnerability
To: undisclosed-recipients:;
Bcc: sxx...@ahbl.org

Hi Team

I am a web app security hunter. I spent some time on your website and found
some vulnerabilities. I see on your website you take security very
passionately.

Tell me will you give me rewards for my finding and responsible
disclosure? if Yes, So tell me where I send those vulnerability reports?
share email address.

Thank you

Good day, I truly hope it treats you awesomely on your side of the screen :)

x Security
--


Is soliciting for money/rewards when the site makes no indication they
offer them a common thing now?

If you want to see a copy of the original message, let me know off list
and I'll send it to you.


-- 
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org / http://www.ahbl.org

Re: Cogent cutting links to Russia?

[Aaron Wendel]

I think you're reading it incorrectly.

The US government and many other countries have imposed sanctions 
against Russia and barred businesses in those countries from doing 
business in Russia.  Cogent is a US based company and, even if it 
operates on foreign jurisdictions through subsidiaries, has issues 
providing services to sanctioned entities.  That's how I read the 
excerpt provided.


Aaron


On 3/4/2022 4:03 PM, Matthew Petach wrote:



On Fri, Mar 4, 2022 at 12:55 PM Martin Hannigan  
wrote:



I would argue they don't have much of a choice:

"The economic sanctions put in place as a result of the invasion
and the increasingly uncertain security situation make it
impossible for Cogent to continue to provide you with service."

I would expect to see others follow suit  if that is the case.



That's an interesting slope to slide along...

I fully understand ISPs disconnecting customers for non-payment; we've
all had to do that at one point or another in our careers, I'm sure.
However, that's generally done *after* the customer has demonstrated
an inability or unwillingness to pay their bills.

This doesn't seem to indicate that any existing invoices have gone
unpaid past their due date, but simply that there is *concern* that a
future bill might go unpaid due to the economic sanctions.

I'm not sure that's a good precedent for a service provider to create;
"we may terminate your service at any point if we suspect that at an
unspecified time in the future, you may become unable to pay future
invoices."

Shades of Minority Report.  We'll imprison you today for a crime we
suspect you will commit in the future.   ^_^;

If and when bills go unpaid, I fully support turning off customers.
I worry about the precedent of disconnecting based on suspicions
of what might happen in the future, however.

Matt


--
====
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: New minimum speed for US broadband connections

[Aaron Porter]

Same issues in NYC. I'm in the bay area burbs and at least once a month get
marketing from AT or Sonic about FTTH that stops 2 doors away. The bonded
DSL alternative is... Functional but a couple times more expensive than my
neighbors pay.

https://arstechnica.com/tech-policy/2020/11/verizon-wiring-up-500k-homes-with-fios-to-settle-years-long-fight-with-nyc/

On Wed, Feb 16, 2022, 10:38 AM Mike Hammett  wrote:

> *nods*
>
> If there's not a fiscal reason to not do it (which USF and other
> give-aways solve), then there's a political reason. Gotta solve that one on
> a case-by-case basis.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Aaron Wendel" 
> *To: *nanog@nanog.org
> *Sent: *Wednesday, February 16, 2022 12:13:52 PM
> *Subject: *Re: New minimum speed for US broadband connections
>
> The reason government incentives exist is because, in a lot of rural
> America, a business case can't be made to connect to Grandma's farm
> that's 10 miles from the nearest splice box.  If you believe that broad
> band is a basic service now, like electricity, then getting Grandma her
> porn is important enough to subsidize.
>
> If I want to run fiber to every home in the 11th larges city with a
> population density of 5,642 people/sq mi, that's an easy case to make
> from a financial perspective.  The issues that come into play are local
> red tape, fees, restrictions, etc.  Compound that with large providers
> agreeing not to overbuild each other and incentives given by said large
> providers to developers and, sometimes, its just not worth it.
>
> Here's an example for you.  North Kansas City, Missouri has FREE gigabit
> fiber to every home in town.  It also has Spectrum (Charter) and AT
> Recently there has been a boom of apartment complexes going up but they
> don't get the free stuff. Why?  Because Spectrum and Charter pay the
> developers to keep the free stuff by assuming internal infrastructure
> costs and/or paying the developments and complexes a kickback for every
> subscriber. Now the FCC says you can't do that but they get around it by
> altering the language in their agreements.
>
> Aaron
>
>
> On 2/16/2022 11:52 AM, Owen DeLong via NANOG wrote:
> >
> >
> >> On Feb 11, 2022, at 13:14 , Josh Luthman
> >>  wrote:
> >>
> >> Because literally every case I've seen along these lines is someone
> >> complaining about the coax connection is "only 100 meg when I pay for
> >> 200 meg". Comcast was the most hated company and yet they factually
> >> had better speeds (possibly in part to their subjectively terrible
> >> customer service) for years.
> >>
> >> >An apartment building could have cheap 1G fiber and the houses
> >> across the street have no option but slow DSL.
> >>
> >> Where is this example?  Or is this strictly hypothetical?
> >
> > There are literally dozens (if not thousands) of such examples in
> > silicon valley alone.
> >
> >> I am not seeing any examples, anywhere, with accurate data, where
> >> it's what most consider to be in town/urban and poor speeds.  The
> >> only one that was close was Jared and I'm pretty sure when I saw the
> >> map I wouldn't consider that in town (could be wrong) but again,
> >> there's gig fiber there now.  I don't remember if he actually got his
> >> CLEC, or why that matters, but there's fiber there now.
> >
> > Pretty sure you would have a hard time calling San Jose “not in town”.
> > It’s literally #11 in the largest 200 cities in the US with a
> > population of 1,003,120 (954,940 in the 2010 census) and a population
> > density of 5,642 people/sq. mile (compare to #4 Houston, TX at
> > 3,632/Sq. Mi.).
> >
> > Similar conditions exist in parts of Los Angeles, #2 on the same list
> > at 3,985,516 (3,795,512 in 2010 census) and 8,499/Sq. Mi.
> >
> > I speak of California because it’s where I have the most information.
> > I’m sure this situation exists in other states as well, but I don’t
> > have actual data.
> >
> > The simple reality is that there are three sets of incentives that
> > utilities tend to chase and neither of them provides for the
> > mezzo-urban and sub-urban parts of America…
> > 1.USF — Mostly supports rural deployments.
> > 2.Extreme High Density — High-Rise apartments in dense arrays, Not
> > areas of town houses, smaller apartment complexes, or single family
> > dwellings.
> > 3.Neighborhoods full of McMansions — Mo

Re: New minimum speed for US broadband connections

[Aaron Wendel]

The reason government incentives exist is because, in a lot of rural 
America, a business case can't be made to connect to Grandma's farm 
that's 10 miles from the nearest splice box.  If you believe that broad 
band is a basic service now, like electricity, then getting Grandma her 
porn is important enough to subsidize.


If I want to run fiber to every home in the 11th larges city with a 
population density of 5,642 people/sq mi, that's an easy case to make 
from a financial perspective.  The issues that come into play are local 
red tape, fees, restrictions, etc.  Compound that with large providers 
agreeing not to overbuild each other and incentives given by said large 
providers to developers and, sometimes, its just not worth it.


Here's an example for you.  North Kansas City, Missouri has FREE gigabit 
fiber to every home in town.  It also has Spectrum (Charter) and AT  
Recently there has been a boom of apartment complexes going up but they 
don't get the free stuff. Why?  Because Spectrum and Charter pay the 
developers to keep the free stuff by assuming internal infrastructure 
costs and/or paying the developments and complexes a kickback for every 
subscriber. Now the FCC says you can't do that but they get around it by 
altering the language in their agreements.


Aaron


On 2/16/2022 11:52 AM, Owen DeLong via NANOG wrote:



On Feb 11, 2022, at 13:14 , Josh Luthman 
 wrote:


Because literally every case I've seen along these lines is someone 
complaining about the coax connection is "only 100 meg when I pay for 
200 meg". Comcast was the most hated company and yet they factually 
had better speeds (possibly in part to their subjectively terrible 
customer service) for years.


>An apartment building could have cheap 1G fiber and the houses 
across the street have no option but slow DSL.


Where is this example?  Or is this strictly hypothetical?


There are literally dozens (if not thousands) of such examples in 
silicon valley alone.


I am not seeing any examples, anywhere, with accurate data, where 
it's what most consider to be in town/urban and poor speeds.  The 
only one that was close was Jared and I'm pretty sure when I saw the 
map I wouldn't consider that in town (could be wrong) but again, 
there's gig fiber there now.  I don't remember if he actually got his 
CLEC, or why that matters, but there's fiber there now.


Pretty sure you would have a hard time calling San Jose “not in town”. 
It’s literally #11 in the largest 200 cities in the US with a 
population of 1,003,120 (954,940 in the 2010 census) and a population 
density of 5,642 people/sq. mile (compare to #4 Houston, TX at 
3,632/Sq. Mi.).


Similar conditions exist in parts of Los Angeles, #2 on the same list 
at 3,985,516 (3,795,512 in 2010 census) and 8,499/Sq. Mi.


I speak of California because it’s where I have the most information. 
I’m sure this situation exists in other states as well, but I don’t 
have actual data.


The simple reality is that there are three sets of incentives that 
utilities tend to chase and neither of them provides for the 
mezzo-urban and sub-urban parts of America…

1.USF — Mostly supports rural deployments.
2.Extreme High Density — High-Rise apartments in dense arrays, Not 
areas of town houses, smaller apartment complexes, or single family 
dwellings.
3.Neighborhoods full of McMansions — Mostly built very recently and 
where the developers would literally pay the utilities to pre-deploy 
in order to boost sales prices.


Outside of those incentives, there’s very little actual deployment of 
broadband improvements, leaving vast quantities of average Americans 
underserved.


Owen





On Fri, Feb 11, 2022 at 4:05 PM Brandon Svec via NANOG 
 wrote:


What is the point of these anecdotes? Surely anyone on this list
with even a passing knowledge of the broadband landscape in the
United States knows how hit or miss it can be.  An
apartment building could have cheap 1G fiber and the houses
across the street have no option but slow DSL.  Houses could have
reliable high speed cable internet, but the office park across
the field has no such choice because the buildout cost is
prohibitively high to get fiber, etc.

There are plenty of places with only one or two choices of
provider too.  Of course, this is literally changing by the
minute as new services are continually being added and upgraded.
*Brandon Svec*



On Fri, Feb 11, 2022 at 12:36 PM Josh Luthman
 wrote:

OK the one example you provided has gigabit fiber though.

On Fri, Feb 11, 2022 at 8:41 AM Tom Beecher
 wrote:

Can you provide examples?


https://www.youtube.com/watch?v=Twe6uTwOyJo_channel=NANOG
<https://www.youtube.com/watch?v=Twe6uTwOyJo_channel=NANOG>

Our good friend Jared could only get 1.5M DSL living just
outside Ann Arbor, MI, so he had to start his own CLEC.

Re: Authoritative Resources for Public DNS Pinging

[Aaron Wendel]

I'd just like to mention that PornHub is always up. (Pun intended)  Ping it.

Aaron


On 2/9/2022 2:43 PM, Tom Beecher wrote:
I mean if you own it, it's your money. But I think I anyone else would 
have a difficult time making a business or technical case to justify 
setting up and maintaining a large scale echo-reply endpoint for... 
what exactly?


On Wed, Feb 9, 2022 at 3:32 PM Lady Benjamin Cannon of Glencoe 
 wrote:


Perhaps owning a (small but global) cloud computing & telecom
company has spoiled me, but it seems like a trivial amount of
resources to me for any moderately sized company let alone a large
tech/telecom like anything you’d have heard of.

-LB

Ms. Lady Benjamin PD Cannon of Glencoe, ASCE
6x7 Networks & 6x7 Telecom, LLC
CEO
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications
company in the world.”
ANNOUNCING: 6x7 GLOBAL MARITIME
<https://alexmhoulton.wixsite.com/6x7networks>

FCC License KJ6FJJ




On Feb 9, 2022, at 12:15 PM, Tom Beecher  wrote:

Side note, am I missing something obvious where I can’t just
have hardware routers strip ICMP, pipe it separately, put 500
VMs behind 4 vLBs and let the world ping the brains out of it?


Seems like a lot of overhead for zero benefit.

On Wed, Feb 9, 2022 at 2:11 PM Lady Benjamin Cannon of Glencoe
 wrote:

ok that’s amazing.

RFC1149 amazing.


Side note, am I missing something obvious where I can’t just
have hardware routers strip ICMP, pipe it separately, put 500
VMs behind 4 vLBs and let the world ping the brains out of it?

Who owns 69.69.69.69 - collab?

How naff is this?

-LB

Ms. Lady Benjamin PD Cannon of Glencoe, ASCE
6x7 Networks & 6x7 Telecom, LLC
CEO
b...@6by7.net
"The only fully end-to-end encrypted global
telecommunications company in the world.”
ANNOUNCING: 6x7 GLOBAL MARITIME
<https://alexmhoulton.wixsite.com/6x7networks>

FCC License KJ6FJJ




On Feb 9, 2022, at 9:38 AM, Jay Hennigan  wrote:

On 2/8/22 23:42, Stephane Bortzmeyer wrote:


The only problem is the less friendly IP address (although
this will
be less and less a problem with IPv6, since
2001:4860:4860:: is
not really friendly).


Fun fact: Someone at Sprint had the same hobby as I did in
the early 1970s. Their website resolves to 2600:: which I
think is rather friendly. :-)

Please don't use it for an IPv6 ping target, thanks.

-- 
Jay Hennigan - j...@west.net

Network Engineering - CCIE #7880
503 897-8550 - WB6RDV






--
====
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: Fiber contractor in Washington state

[Aaron C. de Bruyn via NANOG]

Cascade Networks out of Longview Washington does (or used to do) fiber
installs.
They got bought out by Wave a few years ago, but I think their fiber
division is still active.

https://cni.net/

-A

On Tue, Feb 8, 2022 at 4:50 PM Ross Tajvar  wrote:

> Hi all,
>
> I'm looking for a fiber contractor to trench some fiber on private
> property and then splice it inside. The work will be in Washington state,
> north of Spokane. Does anyone have recommendations? On- and off-list
> welcome.
>
> Thanks,
> Ross
>

Re: Telia is now Arelion

[Aaron C. de Bruyn via NANOG]

On Wed, Jan 19, 2022 at 10:29 AM james.cut...@consultant.com <
james.cut...@consultant.com> wrote:

> As in any other company, the Marketing Department has to find some
> activity to prove their worth.
>

I don't think you realize just how much effort you have to put into finding
a vaguely pronounceable .com domain in today's world.
Marketing probably spent months asking the tech staff to do whois lookups
for them until they found something vaguely marketable.

I'm sure we'll eventually get back to AOL keyword searches and let anyone
register any random TLD they want.  Just email aaron@isawesome.
What?  No .com?
Nope.
What's your website?
Just type the keyword..er...domain 'isawesome' into your browser.

-A

Re: Coverage of the .to internet outage

[Aaron C. de Bruyn via NANOG]

On Thu, Jan 20, 2022 at 10:21 AM Eric Kuhnke  wrote:

> If you're a small pacific island nation state with a limited budget, and a
> working submarine cable, maintaining a SCPC geostationary satellite service
> that might be $20,000 a month (on 36-60 month term) in transponder kHz may
> seem like a very large ongoing expense.
>

Redundancy seems like it could be covered by increasing the cost of a .to
domain.

DNS for .to domains seems to be working just fine, but whois lookups for
.to domains fail with a timeout.

-A

Re: home router battery backup

[Aaron C. de Bruyn via NANOG]

On Mon, Jan 17, 2022 at 11:43 AM Jeff Shultz  wrote:

> BTW, Calix ONTs default to "Disable on battery = on" for the GigE ports -
> it's checkbox in the config to turn that off so they stay up when the power
> is out. Which we do uncheck. Particularly since we've going increasingly
> VOIP and our employees can connect remotely. Sadly, I suspect that trying
> to get a major telco to go in and uncheck that box for you would be the
> equivalent to talking to a wall.
>

My "small" (< ~5,000 customers) ISP won't uncheck that box for me no matter
how much I beg, plead, or offer to bring them snacks for their office.
They keep mumbling stuff about FCC requirements which I suspect is just
handwaving.  Oh well...it's on a generator-protected outlet now.

-A

Re: home router battery backup

[Aaron C. de Bruyn via NANOG]

On Thu, Jan 13, 2022 at 7:41 AM Jay  wrote:

> We consume around 150 watts on DC and generally around 600 watts on AC
> (unless a freezer or air conditioner cycles on).  When the power goes out,
> sometimes we don't immediately notice it!  I think I am living inside a
> giant UPS, and more independance from the Grid is refreshing.
>

*boggles*

I bought one of those power monitors and tossed it on the circuit that goes
into my house.  At *night* when everything is off, I might get down as far
as ~800 watts.
During the day it's more like 2,000-3,500.  If I get the hat-trick (water
heater, central air, and well pump) running at the same time, I can get up
to ~24,000 watts.

The down-side...it's only monitoring the branch that leads to the house.
My office is on a separate branch.

My neighbor pays around $150 every two months on their power bill.
I pay just under ~$260 *every month*.
*sigh*

I definitely notice it when the power goes out.  The sound of UPS relays
and alarms is enough to wake the dead.

-A

Re: home router battery backup

[Aaron C. de Bruyn via NANOG]

On Wed, Jan 12, 2022 at 10:18 AM Andy Ringsmuth  wrote:

> Given that most people barely even know what their home router is, I
> suspect the percentage would be somewhere south of 1 percent. Outside of my
> home, I honestly cannot recall EVER seeing someone’s home using a battery
> backup for their internet infrastructure.
>

Same here.  The only people I've seen that have battery backups for their
home routers are fellow geeks.  I even bought one and shipped it to my
~70-year-old mother...and she just doesn't want to install it.  "Too
complicated".


> I personally do, but of course I (and probably everyone on this list) am
> by no means representative of the population at large in this particular
> area.
>

Same.  My home office has 3 Cyberpower 2500 VA double-conversion UPS units
backed by Champion transfer switches.  Power goes out, and ~45 seconds
later I'm running on generator power.
My local ISP runs out of power well before I do.  Thankfully there's
Starlink.

Short of an asteroid hitting my office, it's highly unlikely I'll ever be
offline. ;)

-A

Re: DOJ files suit to enforce FCC penalty for robocalls

[Aaron C. de Bruyn via NANOG]

My normal test for this is to register a new domain name and leave my whois
info public.

Over the span of 1-2 weeks I will usually get 50-100 calls from people with
a certain accent asking for a  mispronunciation of my name and if I need a
website developed.  Then I forward them over to my spam recording line.

I registered a handful of new domains this week, and I've had less than 5
calls so far.

-A


On Thu, Oct 21, 2021 at 12:13 PM Michael Thomas  wrote:

>
> On 10/21/21 10:57 AM, Sean Donelan wrote:
> >
> > The multi-million dollar fines announced with great fanfaire by the
> > Federal Communication Commission are almost never collected. The FCC
> > doesn't have enforcement authority to collect fines. The FCC usually
> > withholds license renewals until penalties are paid. If the violator
> > doesn't have any FCC licenses (or doesn't care), the FCC is powerless.
> >
> > The FCC refers uncollected penalties to the Department of Justice. In
> > the past, DOJ didn't prioritize uncollected penalties and most fines
> > were never enforced.
> >
> >
> > The Department of Justice Files Suit to Recover $9.9 Million
> > Forfeiture Penalty for Nearly 5,000 Illegally Spoofed Robocalls
> >
> >
> https://www.justice.gov/opa/pr/department-justice-files-suit-recover-forfeiture-penalty-nearly-5000-illegally-spoofed
> >
>
> So has any of the STIR/SHAKEN stuff that was mandated made any
> difference on the ground yet? I assume this is different than what you
> posted about though.
>
> Mike
>
>

Re: massive facebook outage presently

[Aaron C. de Bruyn via NANOG]

It looks like it might take a while according to a news reporter's tweet:

"Was just on phone with someone who works for FB who described employees
unable to enter buildings this morning to begin to evaluate extent of
outage because their badges weren’t working to access doors."

https://twitter.com/sheeraf/status/1445099150316503057?s=20

-A

On Mon, Oct 4, 2021 at 1:41 PM Eric Kuhnke  wrote:

> I am starting to see reports that in ISPs with very large numbers of
> residential users, customers are starting to press the factory-reset
> buttons on their home routers/modems/whatever, in an attempt to make
> Facebook work. This is resulting in much heavier than normal first tier
> support volumes. The longer it stays down the worse this is going to get.
>
>
>
> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan  wrote:
>
>> On 10/4/21 12:11, b...@theworld.com wrote:
>> >
>> > Although I believe it's generally true that if a company appears
>> > prominently in the news it's liable to be attacked I assume because
>> > the miscreants sit around thinking "hmm, who shall we attack today oh
>> > look at that shiny headline!" I'd hate to ascribe any altruistic
>> > motivation w/o some evidence like even a credible twitter post (maybe
>> > they posted that on FB? :-)
>>
>> I personally believe that the outage was caused by human error and not
>> something malicious. Time will tell.
>>
>> However, if you missed the 60 Minutes piece, it was a former employee
>> who spoke out with some rather powerful observations. I don't think that
>> this type of worldwide outage was caused by an outside bad actor. It is
>> certainly within the realm of possibility that it was an inside job.
>>
>> In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>> --
>> Jay Hennigan - j...@west.net
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>
>

Re: Fiber Network Equipment Commercial Norms

[Aaron Wendel]

The building owner has no obligation to the provider.  If it provides no 
value, call them and tell them to remove the equipment if you don't want 
it in your building.


Aaron


On 9/22/2021 11:23 AM, jra...@gmail.com wrote:


A few of the buildings that my firm represents have the local telco’s 
fiber distribution and/or repeater equipment located on the premises. 
My understanding is that when one of these links go down, (we’ve 
occasionally had to interrupt circuit power to do maintenance in a 
building for one reason or another), a local engineering tech always 
comes running to restore the link. The tech has led our maintenance 
staff to believe that these repeaters are an integral part of the 
local ring, which fits my understanding.


When a network operator has equipment located at a third party 
premises, what is the norm for commercial contractual terms regarding 
the siting of that equipment? Any network equipment on site pre-dates 
my client’s ownership of the buildings, and they have no record of any 
agreements or easements governing who is responsible for power, 
maintenance, liability, etc.


My client has no philosophical objection to having the equipment on 
site, but he’s asked why he has had to pay to power and cool this 
equipment for almost 20 years when it serves him no benefit (he is not 
utilizing that company’s services). I figure some of you may be able 
to give me an insight as to what is normal and reasonable. Feel free 
to contact me directly if this message is not suitable for this 
distribution list.


Appreciate the insight,

*Jeff Ray*

O:  (956) 542-3642

C:  (956) 592-2019

jra...@gmail.com

This message has been sent as a part of a discussion between Jeff Ray 
and the intended recipient identified above. Some topics may be 
sensitive and subject to legal privilege, confidentiality, or other 
non-disclosure agreement. Should you receive this message by mistake, 
we would be most grateful if you informed us that the message has been 
sent to you. In that case, we also ask that you delete this message 
from your mailbox, and do not forward or speak of it (or its contents) 
to anyone else. Thank you for your cooperation and understanding.




--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: IPv6 woes - RFC

[Aaron C. de Bruyn via NANOG]

On Sat, Sep 4, 2021 at 9:36 PM Mark Tinka  wrote:

> Supporting the routing and forwarding of IP addresses is just about the
> most basic thing any ISP should do.
>
> If that is low on their to-do list, what else could they possibly be doing?
>

Counting all the profit they make from a captive audience with no
competition? ;)

-A

Re: Reminder: Never connect a generator to home wiring without transfer switch

[Aaron C. de Bruyn via NANOG]

During the February 2021 storm that swept through the US, power got knocked
out on my rural street due to a tree coming down and taking out a pole.

While they were waiting for a few more trucks to arrive with a replacement
pole, I got to ask them a few questions.  They said it's standard practice
for them to ground on both sides exactly for the reason that someone might
accidentally connect a generator.  They open the nearest switch on the
upstream side, test to make sure the line is dead, install grounds on all
the wires, then test the downstream side and attach grounds to all the
wires, effectively making the work zone an isolated segment.

I doubt it's "if you follow every step perfectly at all times and never
make a mistake".
There are usually redundancies built-in when it comes to safety.  i.e.
what's the point of installing grounds on the upstream side if you have the
switch open?  If the lines are de-energized, why wear gloves?  If you're
doing all that, why carry an AED?

-A

On Mon, Aug 30, 2021 at 10:19 AM Warren Kumari  wrote:

>
>
> On Mon, Aug 30, 2021 at 12:47 PM Aaron C. de Bruyn via NANOG <
> nanog@nanog.org> wrote:
>
>> I've been following the thread.
>> If I'm dumb enough to back feed through the transformer into the
>> downstream side of the downed line, how is it going to be a problem if
>> linemen are grounding the phases on *both sides* of the work area.
>>
>
> I suspect that there is a non-zero amount of "in an ideal, perfect world,
> when all of the wires are simply lines on a piece of paper, and you can
> look at them from the comfort of your office chair, this is easy" - but, in
> the real world, linesmen are rushing about and trying to get the lights
> back on, cut through the big ash tree that is wedged between the oak and
> the pole, etc. Even the nice idea of "well, just take the conductos and tie
> 'em to ground" means that you need to go trudging through hedges and
> vegetation and tree limbs and lions and tigers and bears, often while it is
> pissing down with rain or baking hot.
>
> I guess I'm missing how we've moved from the "some people are putting
> their lives on the line, let's try to make their life less dangerous" into
> a "well... if they simply followed these set of steps perfectly at all
> times, and never made a mistake they'd be fine."
> This is NANOG -- I'm sure that we've all followed a set of steps perfectly
> and still managed to redistribute BGP into the IGP, or apply an ACL and
> lock ourselves out of a box, or types "show run" and watched the router
> randomly reboot. Now consider this, but with the added drama of potentially
> ending up dead...
>
> W
>
>
>> That's what Ben seemed to be implying.
>>
>> -A
>>
>> On Mon, Aug 30, 2021 at 9:09 AM Mel Beckman  wrote:
>>
>>> Aaron,
>>>
>>> If you read back in this thread (using the NANOG mailing list archive),
>>> you’ll find this has been explained in great detail. In a nutshell, phase
>>> grounding won’t help if a generator is energized from the customer end, and
>>> this technique was discontinued in the 1970s due to the many deaths that
>>> resulted.
>>>
>>>  -mel
>>>
>>> On Aug 30, 2021, at 9:02 AM, Aaron C. de Bruyn via NANOG <
>>> nanog@nanog.org> wrote:
>>>
>>> 
>>> On Mon, Aug 30, 2021 at 7:35 AM Lady Benjamin Cannon of Glencoe, ASCE <
>>> l...@6by7.net> wrote:
>>>
>>>> Yes, this is a real and dangerous problem.  Today.  Even with grounding
>>>> I’m afraid.  Source: I’ve been working in an engineering capacity for 27
>>>> years and I have the license you’d need to build a nuclear power plant.
>>>>
>>>
>>> Would you care to educate me on this?
>>> If you ground the phases on both sides of the work-site, how are you
>>> going to end up being a better path to ground?
>>>
>>> -A
>>>
>>>
>
> --
> The computing scientist’s main challenge is not to get confused by the
> complexities of his own making.
>   -- E. W. Dijkstra
>

Re: Reminder: Never connect a generator to home wiring without transfer switch

[Aaron C. de Bruyn via NANOG]

I've been following the thread.
If I'm dumb enough to back feed through the transformer into the downstream
side of the downed line, how is it going to be a problem if linemen are
grounding the phases on *both sides* of the work area.
That's what Ben seemed to be implying.

-A

On Mon, Aug 30, 2021 at 9:09 AM Mel Beckman  wrote:

> Aaron,
>
> If you read back in this thread (using the NANOG mailing list archive),
> you’ll find this has been explained in great detail. In a nutshell, phase
> grounding won’t help if a generator is energized from the customer end, and
> this technique was discontinued in the 1970s due to the many deaths that
> resulted.
>
>  -mel
>
> On Aug 30, 2021, at 9:02 AM, Aaron C. de Bruyn via NANOG 
> wrote:
>
> 
> On Mon, Aug 30, 2021 at 7:35 AM Lady Benjamin Cannon of Glencoe, ASCE <
> l...@6by7.net> wrote:
>
>> Yes, this is a real and dangerous problem.  Today.  Even with grounding
>> I’m afraid.  Source: I’ve been working in an engineering capacity for 27
>> years and I have the license you’d need to build a nuclear power plant.
>>
>
> Would you care to educate me on this?
> If you ground the phases on both sides of the work-site, how are you going
> to end up being a better path to ground?
>
> -A
>
>

Re: Reminder: Never connect a generator to home wiring without transfer switch

[Aaron C. de Bruyn via NANOG]

On Mon, Aug 30, 2021 at 7:35 AM Lady Benjamin Cannon of Glencoe, ASCE <
l...@6by7.net> wrote:

> Yes, this is a real and dangerous problem.  Today.  Even with grounding
> I’m afraid.  Source: I’ve been working in an engineering capacity for 27
> years and I have the license you’d need to build a nuclear power plant.
>

Would you care to educate me on this?
If you ground the phases on both sides of the work-site, how are you going
to end up being a better path to ground?

-A

Re: An update on the AfriNIC situation

[Aaron Wendel]

I suppose people who wanted to take a side could also block traffic to 
and from Cloud Innovations IP blocks.



On 8/27/2021 10:36 AM, Bill Woodcock wrote:

As many of you are aware, AfriNIC is under legal attack by Heng Lu / “Cloud 
Innovation.”

John Curran just posted an excellent summary of the current state of affairs 
here:


https://teamarin.net/2021/08/27/afrinic-and-the-stability-of-the-internet-number-registry-system/

If, like me, you feel like chipping in a little bit of money to help AfriNIC 
make payroll despite Heng having gotten their bank accounts frozen, some of the 
African ISP associations have put together a fund, which you can donate to here:

https://www.tespok.co.ke/?page_id=14001

It’s an unfortunate situation, but the African Internet community has really 
pulled together to defend themselves, and they’ve got a lot less resources than 
most of us do.

-Bill

Re: netflow in the core used for surveillance

[Aaron Wendel]

You don't know that I don't know that.


On 8/25/2021 4:32 PM, Paul Ebersman wrote:

randy> 
https://www.vice.com/en/article/jg84yy/data-brokers-netflow-data-team-cymru

randy> at, comcast, ... zayo, please tell us you do not do this.


aaron> You know they do.

No, you don't know that.

The above all certainly collect this info. Not all sell it to anyone who
asks.

Re: netflow in the core used for surveillance

[Aaron Wendel]

You know they do.

On 8/25/2021 4:13 PM, Randy Bush wrote:

https://www.vice.com/en/article/jg84yy/data-brokers-netflow-data-team-cymru

used to get dissidents, activists, and journos killed

at, comcast, ... zayo, please tell us you do not do this.

randy

Re: Can somebody explain these ransomwear attacks?

[Aaron C. de Bruyn via NANOG]

On Fri, Jun 25, 2021 at 10:43 AM Tom Beecher  wrote:

> Incompetent insurance companies combined with incompetent IT staff and
>> under-funded IT departments are the nexus of the problem.
>>
>
> Nah, it's even simpler. It's just dollars all around. Always is.
>

Agreed.


> From this company's point of view, the cost to RECOVER from the problems
> is so much smaller than it would be to prevent the problems from happening
> to begin with, so they are happy to let you guys handle it. From the
> insurance company's point of view, they are collecting premiums, but no
> claims are being filed, so they have no incentive to do anything
> differently.
>

I'm sure that'll change drastically if either of these conditions are true:
* A claim is filed
* An audit is required
* Ransomware surges throughout 2021 and payouts go through the roof

I think it's reasonable to expect at least one of those things will happen
in the next year.

-A

>

Re: Can somebody explain these ransomwear attacks?

[Aaron C. de Bruyn via NANOG]

On Fri, Jun 25, 2021 at 5:28 AM Jim  wrote:

> Big problem that with organizations' existing Disaster Recovery DR methods
> --
> the time and cost to recovery from any event including downtime will
> be some amount.. likely a high one,
> and criminals' ransom demands will presumably be set as high a price
> as they think they can get --
> but still orders of magnitudes less than cost to recover / repair /
> restore, and the downtime may be less.
>

I think you're right.  DR methods are a *huge* part of the problem.
I manage DR systems for a number of companies including a large unnamed
healthcare provider.
A year ago they were still running Exchange 2007.  No, that's not a typo.
Cryptolocker strolled right into the network via file attachment and
somehow made it past the non-existent 3rd-party AV software that totally
wasn't integrated into Exchange because it cost too much.
It spread across the network and started encrypting around 1 AM on a Friday
morning.
Due to the way this particular strain worked, it missed several of the
monitoring tools that would have alerted my company to the massive file
encryption that was happening and it managed to completely encrypt 21
offices and all their patient data.
At 6 AM my monitoring system alerted me to a problem.  By about 6:30 I
realized the scope of the problem, disabled all the site-to-site VPNs,
dropped the 1 or 2 infected workstations off the network and the encryption
stopped.
We do local snapshots every 15 minutes, local backups twice daily, local
disconnected backups several times per week, and off-site write-only
backups multiple times per day.
After I figured out when cryptolocker launched, I ran a few commands from
our config management server and had every office restored and running in
about 28 minutes and the internal techs for the company were dispatched to
swap out the infected workstations.

The first rule I follow is: Windows *never* touches bare metal.
I amended that last year to: Windows *never* touches bare metal, including
workstations.

People *really* need to work on their backups and DR plans.  You don't need
some expensive 3rd-party cloud solution coupled with expensive VMWare
licenses to do it.

The other part of the problem is the insurance companies.
It might surprise you to learn that particular company has been
cryptolocker'd 8 times in the last 15 years.  They've never lost more than
a few minutes of data and recovery times are measured in minutes.
This line has literally been thrown around a few times: "We don't need to
spend $xxx,xxx to upgrade to current software versions.  We have a
$5,000,000 cyber insurance policy."

The insurance company issued the policy after *port scanning* their public
IPs and finding no ports open.  Our only 'ding' we got was that the routers
responded to pings and the insurance company thought they shouldn't.
Insurance failed to do any sort of competent audit (i.e. NIST 800-171).  If
they did, they would have found the techs "solve" problems by making people
local admins or domain admins and that their primary line-of-business app
actually requires 'local admin' to run 'properly'.

While they finally replaced Exchange 2007 in 2020 by switching to GMail
(not for security, but because it made work-from-home easier), they still
run about 1/3 of their systems on Windows 7 with a few Windows 8 and 8.1
machines here and there.  They even still have 2 Windows XP machines.
Their upgrade policy is currently "If the machine dies, you can replace it
with something newer".  Their oldest machine is around 15 years old.

Incompetent insurance companies combined with incompetent IT staff and
under-funded IT departments are the nexus of the problem.

-A

Re: Tier1 BGP filter generation data sources & frequency

[Aaron Atac via NANOG]

Peeringdb mostly.

Otherwise, onestep.net has some but not all.

whois when in doubt or email their noc.

-Aaron


May 21, 2021, 16:40 by clin...@scripty.com:

> Is there any compiled information for Tier1 providers on the supported BGP 
> filter generation data sources and frequency? 
>
> This is what I have been able to determine so far: 
> - TATA AS6453:   IRR and RPKI ROAs 
> (http://lg.as6453.net/doc/cust-routing-policy.html)
> - Cogent AS174: unknown
> - NTT 2914:   IRR, ARIN WHOIS OriginAS,  NIC.br whois, RPKI ROAs   
> (https://www.gin.ntt.net/support-center/policies-procedures/routing/)
> - Lumen AS3356:   IRR
> - Telia AS1299:   IRR
>
> TATA is going to deprecate new RADB, NTTCOM, and ALTDB route objects starting 
> Aug 15, 2021 and I was hoping that more providers would add RPKI ROAs as a 
> data source for BGP filter generation.Supporting RPKI ROAs would mean 
> that you don't have to create both IRR route objects and RPKI ROAs for each 
> IP block. 
>
> --
> Clinton Work
>

Re: 10 years from now... (was: internet futures)

[Aaron C. de Bruyn via NANOG]

On Mon, Mar 29, 2021 at 11:39 AM Matt Erculiani 
wrote:

> I think the best way to think about what 10 years from now will look like
> is to compare 10 years ago to the present:
> https://mailman.nanog.org/pipermail/nanog/2011-April/thread.html
>

Multi-homing your DSL connection?
I can't wait to multi-home my 10x10 array of StarLink satellites in a few
years...

-A

Re: Famous operational issues

[Aaron C. de Bruyn via NANOG]

All these stories remind me of two of my own from back in the late 90s.
I worked for a regional ISP doing some network stuff (under the real
engineer), and some software development.

Like a lot of ISPs in the 90s, this one started out in a rental house.
Over the months and years rooms were slowly converted to host more and more
equipment as we expanded our customer base and presence in the region.
If we needed a "rack", someone would go to the store and buy a 4-post metal
shelf [1] or...in some cases the dump to see what they had.

We had one that looked like an oversized filing cabinet with some sort of
rails on the sides.  I don't recall how the equipment was mounted, but I
think it was by drilling holes into the front lip and tapping the screws
in.  This was the big super-important rack.  It had the main router that
connected lines between 5 POPs around the region, and also several
connections to Portland Oregon about 60 miles away.  Since we were
making tons of money, we decided we should update our image and install
real racks in the "bedroom server room".  It was decided we were going to
do it with no downtime.

I was on the 2-man team that stood behind and in front of the rack with
2x4s dead-lifting them as equipment was unscrewed and lowered onto the
boards.  I was on the back side of the rack.  After all the equipment was
unscrewed, someone came in with a sawzall and cut the filing cabinet thing
apart.  The top half was removed and taken away, then we lifted up on the
boards and the bottom half was slid out of the way.  The new rack was
brought in, bolted to the floor, and then one by one equipment was taken
off the pile we were holding up with 2x4s, brought through the back of the
new rack, and then mounted.

I was pleasantly surprised and very relieved when we finished moving the
big router, several switches, a few servers, and a UPS unit over to the new
rack with zero downtime.  The entire team cheered and cracked beers.  I
stepped out from behind the rack...
...and snagged the power cable to the main router with my foot.  I don't
recall the Cisco model number after all this time...but I do remember the
excruciating 6-8 minutes it took for the damn thing to reboot, and the
sight of the 7 PRI cards in our phone system almost immediately jumping
from 5 channels in-use to being 100% full.

It's been 20 years, but I swear my arms are still sore from holding all
that equipment up for ~20 minutes, and I always pick my feet up very slowly
when I'm near a rack. ;)

The second story is a short one from the same time period.  Our POPs
consisted of the afore-mentioned 4-post metal shelves stacked with piles of
US Robotics 56k modems [2] stacked on top of each other.  They were wired
back to some sort of serial box that was in-turn connected to an ISA card
stuck in a Windows NT 4 server that used RADIUS to authenticate sessions
with an NT4 server back at the main office that had user accounts for all
our customers.  Every single modem had a wall-wart power brick for power,
an RJ11 phone line, and a big old serial cable.  It was an absolute rats
nest of cables.  The small POP (which I think was a TuffShed in someone's
yard about 50 feet from the telco building) was always 100 degrees--even in
the dead of winter.

One year we made the decision to switch to 3Com Total Control Chassis with
PRI cards.  The cut-over was pretty seamless and immediately made shelves
stacked full of hundreds of modems completely useless.  As we started
disconnecting modems with the intent of selling them for a few bucks to
existing customers who wanted to upgrade or giving them to new customers to
get them signed up, we found a bunch of the stacks of modems had actually
melted together due to the temps.  That explained the handful of numbers in
the hunt group that would just ring and ring with no answer.  In the end we
went from a completely packed 10x20 shed to two small 3Com TCH boxes packed
with PRI cards and a handful of PRI cables with much more normal
temperatures.

I thoroughly enjoyed the "wild west" days of the internet.

If Eric and Dan are reading this, thanks for everything you taught me about
networking, business, hard work, and generally being a good person.

-A

[1] -
https://www.amazon.com/dp/B01D54TICS/ref=redir_mobile_desktop?_encoding=UTF8=Pe4xuew1D1PkrRA9cq8Cdg_cr_id=5048111780901_rd_plhdr=t_rd_r=4d9e3b6b-3360-41e8-9901-d079ac063f03_rd_w=uRxXq_rd_wg=CDibq_=sbx_be_s_sparkle_td_asin_0_img

[2] - https://www.usr.com/products/56k-dialup-modem/usr5686g/



On Tue, Feb 16, 2021 at 11:39 AM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two 

Re: Texas internet connectivity declining due to blackouts

[Aaron C. de Bruyn via NANOG]

It might not be an easy fix in the moment, but in the long run, buy a
generator and install a propane tank.
When power prices spike to insane levels like this, just flip your transfer
switch over and run off propane.
When utility power becomes cheaper, switch back to the grid.

Maybe some sort of Raspberry Pi to monitor the current prices and do the
transfer automatically.  (language warning:
https://www.youtube.com/watch?v=gz7IPTf1uts)

Protip: If you're blacked out, it doesn't matter what the price of power is.

-A

On Wed, Feb 17, 2021 at 8:47 AM John Sage  wrote:

> On 2/17/21 8:07 AM, Sean Donelan wrote:
> >
> >
> > On Wed, 17 Feb 2021, Andy Ringsmuth wrote:
> >> Not sure where you’re finding those numbers but I believe they are not
> >> accurate.
> >
> > U.S. Energy Information Administration (part of the Department of Energy)
> >
> >
> https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a
>
> This article is an interest description of Texas electricity pricing for
> one provider and for the market in general:
>
> "Some retail power companies in Texas are making an unusual plea to
> their customers amid a deep freeze that has sent electricity prices
> skyrocketing: Please, leave us.
>
> Power supplier, Griddy, told all 29,000 of its customers that they
> should switch to another provider as spot electricity prices soared to
> as high as $9,000 a megawatt-hour. Griddy’s customers are fully exposed
> to the real-time swings in wholesale power markets, so those who don’t
> leave soon will face extraordinarily high electricity bills."
>
> The catch:
>
> "Hector Torres, an energy trader in Texas, who is a Griddy customer
> himself, said he tried to switch services over the long weekend but
> couldn’t find a company willing to take him until Wednesday, when the
> weather is forecast to turn warmer."
>
>
> https://www.dallasnews.com/business/energy/2021/02/16/electricity-retailer-griddys-unusual-plea-to-texas-customers-leave-now-before-you-get-a-big-bill/
>
>
>
> - John
> --
>
>

Re: Alexandria Ocasio-Cortez' Office is on NANOG?? Or, what is the policy about sharing email offlist?

[Aaron C. de Bruyn via NANOG]

On Mon, Jan 18, 2021 at 10:20 AM Anne P. Mitchell, Esq. 
wrote:

> And either way, what is the policy about forwarding list email to someone
> who is not on the list?
>

If you are posting to NANOG under the impression that your email will only
be seen by network engineers and that it will never be "leaked" off-list to
the public, I have deal for you involving a few billion shillings I need to
smuggle out of Kenya...you can keep 10%, and I just need your routing
info...

-A

Re: Parler

[Aaron C. de Bruyn via NANOG]

Maybe read Holmes' dissent where he uses the phrase "fire in a crowded
theater" or at least listen to the cliff notes:
https://www.popehat.com/2018/06/28/make-no-law-episode-seven-fire-in-a-crowded-theater/
.

-A

On Sun, Jan 10, 2021 at 2:59 PM Jay Hennigan  wrote:

> On 1/10/21 13:50, Rod Beck wrote:
>
> > As a big fan of the 1st amendment, but someone deeply appalled by the
> > riot last week and keenly aware of how social media are letting the mud
> > to the surface, I am very perplexed how to reconcile free speech and the
> > garbage flowing through our social streets.
>
> The first amendment deals with the government passing laws restricting
> freedom of speech. It has nothing to do with to whom AWS chooses to sell
> their services. It is also not absolute (fire, crowded theater, etc.)
>
> Has anyone seen a rabbit? We've traveled quite a way down the rabbit hole.
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>

Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study

[Aaron C. de Bruyn via NANOG]

On Mon, Jan 4, 2021 at 7:11 PM Billy Crook  wrote:

> On Fri, Jan 1, 2021 at 4:13 PM Matt Hoppes <
> mattli...@rivervalleyinternet.net> wrote:
>
>> Just give users the ability to select what categories/severities they
>> want to see, so I don't get disrupted every time there's a scary rain storm
>> coming or some divorcee is behind on child-support.
>
>
Yesterday I was mildly indifferent.
Today, after receiving SIX zarking Amber alerts between 8 PM and 11 PM
local time, I suddenly have a strong opinion.
Talk about alert fatigue.  The sixth alert I received could have been for
the world ending.  I still wouldn't have looked at my phone.

Thankfully I can adjust the default setting to disable everything except
"presidental emergency alerts"...whatever that is.

As long as I can turn it off completely, I'm fine with people baking that
crap into their tech.

I still want my wired Nest smoke alarm to be able to pick up NWS alerts
though.

-A

Re: 10g residential CPE

[Aaron Wendel]

It does have wireless.  That doesn't prevent people from trying to use 
their old equipment in addition. ("My dad's uncle's cousin's former 
roommate works in IT and told me I just needed to plug my old router 
into your new router.")


On 12/29/2020 10:53 AM, Michael Thomas wrote:


On 12/29/20 8:42 AM, Aaron Wendel wrote:
Oh, we still get calls about speed issues. It's always wonderful when 
someone puts their own 10 year old Linksys WRT54G and double NATs 
behind our CPE then sends in a speed test wondering why they're only 
getting 10Mbits on their Gbit line.  We get those ALL the time. :)


Does your CPE not have wireless? If it's double NAT'ing it's at least 
a router. If it doesn't have wireless, wouldn't it be cheaper to add 
it so you don't get the support calls?


Mike



--
====
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: 10g residential CPE

[Aaron Wendel]

The majority of our customers are still on Brocade MLXs.  We're in the 
process of upgrading all our equipment to Arista switches to accommodate 
the increased demand for 40G and 100G ports as well as implement 400G ports.


Aaron


On 12/29/2020 3:33 AM, Jonathon Exley wrote:

Hi Aaron,

Just out of interest, what switch gear are you using? You must have a 
pretty good cost per port.


Jonathon.

On 29/12/2020 9:38 AM, Aaron Wendel  wrote:
We prioritize calls based on severity.  If both Google and Grandma call
and say they have a cut then we have people to service both at the same
time.  If Google, Century Link, Verizon, AT and Grandma all call then
Grandma gets to wait a day.  That being the case, it's not dependent on
revenue. Emergency Services (911 and Police radio feeds) gets #1
priority even though they're non-paying.

But yes, in extreme situations the residential customers would be
delayed to service the paying customers.  We do have people cross
trained from other parts of our businesses so we can allocate internally
in emergencies.  In almost a decade though I can't think of a situation
where someone had to wait for service because we didn't have the
resources to service them.

Aaron


On 12/28/2020 2:02 PM, Mel Beckman wrote:
> Darin,
>
> Surely you at least give the paying customers priority over the
> non-paying? It’s one thing to say “I have to write paychecks no matter
> what”. It’s another to say “I’ll give away my support to free
> customers AND degrade support for paying customers as a result.” Your
> tech support guy “walking Grandma through getting her email” is
> necessarily not accessible for the duration to paying customers.
>
> This means your staffing must be large enough to never have any
> queuing, or you’re giving away your paying customers' time to
> non-paying customers. Neither approach is scalable in a competitive
> business environment, because SOMEBODY is paying for all those
> resources, and if it’s your customers, they will buy elsewhere. Your
> approach only work until you run out of other people’s money.
>
>   -mel
>
>> On Dec 28, 2020, at 11:50 AM, Baldur Norddahl
>> mailto:baldur.nordd...@gmail.com>> wrote:
>>
>> I applaud your commitment to helping your local community. Just want
>> to point out that this is a charity because it does not scale. Nobody
>> could build out a FTTH network and make it free as a business case.
>> But there are plenty of people that made a network for their
>> neighbors and provided that for free. Maybe a person had a commercial
>> fiber to his home and thought he could just as well share it. This
>> might be on a bigger scale but it is the same.
>>
>> Regards,
>>
>> Baldur
>>
>>
>> On Mon, Dec 28, 2020 at 8:27 PM Aaron Wendel
>> mailto:aa...@wholesaleinternet.net>> 
wrote:

>>
>> Darin,
>>
>> Our business support and residential support is the same
>> department.  I
>> have to pay those people to be in the office either way so it
>> doesn't
>> cost me any "more" to provide support for the residences. Yes,
>> walking
>> Grandma through getting her email can sometimes be a chore but that
>> person is on the payroll whether he/she is helping Grandma or
>> sitting
>> there chatting with his/her co-worker.  If we dumped all the
>> residential
>> customers we would still have the same cost structure we do now.
>>
>> Again, it's been free for the last 7 years at this point.  I've
>> never
>> been one to really do what I "should" anyway.
>>
>> Aaron
>>
>>
>> On 12/28/2020 11:48 AM, Darin Steffl wrote:
>> > Aaron,
>> >
>> > The "Free" service doesn't cover your cost of support which is
>> much
>> > higher for residential than any business customer. Our residential
>> > customers call at least 15x more often compared to business
>> customers
>> > compared on a 1:1 ratio.
>> >
>> > I honestly can't fathom providing free residential service
>> because we
>> > make enough money on the business side of things. You should be
>> > charging something, at least $20-30 per month.
>> >
>> > On Mon, Dec 28, 2020 at 11:15 AM Aaron Wendel
>> > > <mailto:aa...@wholesaleinternet.net>
>> <mailto:aa...@wholesaleinternet.net
>> <mailto:aa...@wholesaleinternet.net>>> wrote:
>> >
>> >     The $300 covers the equipment and the time to send someone
>> out to a
>> >     house to install it.  If $300 is too much you can pay in 12
>> >     installments
>> >     of $25.
>> >
>> >     The TIK al

Re: 10g residential CPE

[Aaron Wendel]

Oh, we still get calls about speed issues. It's always wonderful when 
someone puts their own 10 year old Linksys WRT54G and double NATs behind 
our CPE then sends in a speed test wondering why they're only getting 
10Mbits on their Gbit line.  We get those ALL the time. :)


On 12/29/2020 1:28 AM, Mark Tinka wrote:



On 12/29/20 04:41, Keith Medcalf wrote:

Are you sure that is not related to "residential services" being of a 
generally lower quality than business services?  It has been my 
experience that shoddy service generates higher need for "support" 
than does "non-shoddy" service.  In this regard, the price for 
"business" services should be less than "residential service" by a 
couple of orders of magnitude since it costs orders of magnitude more 
money to "support" shoddy services than non-shoddy services.


Considering that Aaron said 98% of their residential customers are on 
the free plan, and that they use Active-E with every 1Gbps customer 
getting a proper switch port, I'd hazard the bulk of their support 
queries to be non-techie customers needing software support (grandma, 
et al), or fibres being cut.


It wouldn't seem like they'd be getting calls about "speed" issues, 
which are most annoying ones :-).


Mark.


--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: 10g residential CPE

[Aaron Wendel]

We prioritize calls based on severity.  If both Google and Grandma call 
and say they have a cut then we have people to service both at the same 
time.  If Google, Century Link, Verizon, AT and Grandma all call then 
Grandma gets to wait a day.  That being the case, it's not dependent on 
revenue. Emergency Services (911 and Police radio feeds) gets #1 
priority even though they're non-paying.


But yes, in extreme situations the residential customers would be 
delayed to service the paying customers.  We do have people cross 
trained from other parts of our businesses so we can allocate internally 
in emergencies.  In almost a decade though I can't think of a situation 
where someone had to wait for service because we didn't have the 
resources to service them.


Aaron


On 12/28/2020 2:02 PM, Mel Beckman wrote:

Darin,

Surely you at least give the paying customers priority over the 
non-paying? It’s one thing to say “I have to write paychecks no matter 
what”. It’s another to say “I’ll give away my support to free 
customers AND degrade support for paying customers as a result.” Your 
tech support guy “walking Grandma through getting her email” is 
necessarily not accessible for the duration to paying customers.


This means your staffing must be large enough to never have any 
queuing, or you’re giving away your paying customers' time to 
non-paying customers. Neither approach is scalable in a competitive 
business environment, because SOMEBODY is paying for all those 
resources, and if it’s your customers, they will buy elsewhere. Your 
approach only work until you run out of other people’s money.


  -mel

On Dec 28, 2020, at 11:50 AM, Baldur Norddahl 
mailto:baldur.nordd...@gmail.com>> wrote:


I applaud your commitment to helping your local community. Just want 
to point out that this is a charity because it does not scale. Nobody 
could build out a FTTH network and make it free as a business case. 
But there are plenty of people that made a network for their 
neighbors and provided that for free. Maybe a person had a commercial 
fiber to his home and thought he could just as well share it. This 
might be on a bigger scale but it is the same.


Regards,

Baldur


On Mon, Dec 28, 2020 at 8:27 PM Aaron Wendel 
mailto:aa...@wholesaleinternet.net>> wrote:


Darin,

Our business support and residential support is the same
department.  I
have to pay those people to be in the office either way so it
doesn't
cost me any "more" to provide support for the residences. Yes,
walking
Grandma through getting her email can sometimes be a chore but that
person is on the payroll whether he/she is helping Grandma or
sitting
there chatting with his/her co-worker.  If we dumped all the
residential
customers we would still have the same cost structure we do now.

Again, it's been free for the last 7 years at this point.  I've
never
been one to really do what I "should" anyway.

Aaron


On 12/28/2020 11:48 AM, Darin Steffl wrote:
> Aaron,
>
> The "Free" service doesn't cover your cost of support which is
much
> higher for residential than any business customer. Our residential
> customers call at least 15x more often compared to business
customers
> compared on a 1:1 ratio.
>
> I honestly can't fathom providing free residential service
because we
> make enough money on the business side of things. You should be
> charging something, at least $20-30 per month.
>
> On Mon, Dec 28, 2020 at 11:15 AM Aaron Wendel
> mailto:aa...@wholesaleinternet.net>
<mailto:aa...@wholesaleinternet.net
<mailto:aa...@wholesaleinternet.net>>> wrote:
>
>     The $300 covers the equipment and the time to send someone
out to a
>     house to install it.  If $300 is too much you can pay in 12
>     installments
>     of $25.
>
>     The TIK alone costs us about $250.
>
>     Aaron
>
>
>     On 12/27/2020 5:04 AM, Mark Tinka wrote:
>     >
>     >
>     > On 12/26/20 20:48, Darin Steffl wrote:
>     >
>     >> Aaron,
>     >>
>     >> One simple question. Why on earth would you offer free
internet
>     >> service? How and why? Your site show 1 Gig symmetrical
for free
>     when
>     >> you should be a minimum of $65 per month to be competitive.
>     >
>     > They also ask for no monthly fee after a single payment
of US$300.
>     >
>     > Considering the 2Gbps package costs US$49.95, you'd guess
they'd
>     value
>     > the 1Gbps service at, say US$27/month, give or take.
>     >
>     > So that US$300 provide

Re: 10g residential CPE

[Aaron Wendel]

We still build when needed. We're in the process of building to 700 new 
apartments so we can provide them with free service.  We're actually 
pulling 576 strands into the basement of one building to backhaul each 
apartment to it's own switch port in the new hut we just deployed to 
service that new development.  (we don't use a PON system.  Everyone has 
a dedicated switch port.)  Also, keep in mind that this isn't all we 
do.  This is a very small part of a much bigger pie.  So I agree with 
you.  If this was it then it would make no sense.  When you look at all 
the pieces together it makes perfect sense.


Aaron


On 12/28/2020 1:50 PM, Baldur Norddahl wrote:
I applaud your commitment to helping your local community. Just want 
to point out that this is a charity because it does not scale. Nobody 
could build out a FTTH network and make it free as a business case. 
But there are plenty of people that made a network for their neighbors 
and provided that for free. Maybe a person had a commercial fiber to 
his home and thought he could just as well share it. This might be on 
a bigger scale but it is the same.


Regards,

Baldur


On Mon, Dec 28, 2020 at 8:27 PM Aaron Wendel 
mailto:aa...@wholesaleinternet.net>> wrote:


Darin,

Our business support and residential support is the same
department.  I
have to pay those people to be in the office either way so it doesn't
cost me any "more" to provide support for the residences. Yes,
walking
Grandma through getting her email can sometimes be a chore but that
person is on the payroll whether he/she is helping Grandma or sitting
there chatting with his/her co-worker.  If we dumped all the
residential
customers we would still have the same cost structure we do now.

Again, it's been free for the last 7 years at this point. I've never
been one to really do what I "should" anyway.

Aaron


On 12/28/2020 11:48 AM, Darin Steffl wrote:
> Aaron,
>
> The "Free" service doesn't cover your cost of support which is much
> higher for residential than any business customer. Our residential
> customers call at least 15x more often compared to business
customers
> compared on a 1:1 ratio.
>
> I honestly can't fathom providing free residential service
because we
> make enough money on the business side of things. You should be
> charging something, at least $20-30 per month.
>
> On Mon, Dec 28, 2020 at 11:15 AM Aaron Wendel
> mailto:aa...@wholesaleinternet.net>
<mailto:aa...@wholesaleinternet.net
<mailto:aa...@wholesaleinternet.net>>> wrote:
>
>     The $300 covers the equipment and the time to send someone
out to a
>     house to install it.  If $300 is too much you can pay in 12
>     installments
>     of $25.
>
>     The TIK alone costs us about $250.
>
>     Aaron
>
>
>     On 12/27/2020 5:04 AM, Mark Tinka wrote:
>     >
>     >
>     > On 12/26/20 20:48, Darin Steffl wrote:
>     >
>     >> Aaron,
>     >>
>     >> One simple question. Why on earth would you offer free
internet
>     >> service? How and why? Your site show 1 Gig symmetrical
for free
>     when
>     >> you should be a minimum of $65 per month to be competitive.
>     >
>     > They also ask for no monthly fee after a single payment of
US$300.
>     >
>     > Considering the 2Gbps package costs US$49.95, you'd guess
they'd
>     value
>     > the 1Gbps service at, say US$27/month, give or take.
>     >
>     > So that US$300 provides a bit of coverage, perhaps 1 year,
in which
>     > time they'd have likely upgraded the customer.
>     >
>     > Mark.
>
>     --
>  
>     Aaron Wendel
>     Chief Technical Officer
>     Wholesale Internet, Inc. (AS 32097)
>     (816)550-9030
> http://www.wholesaleinternet.com
<http://www.wholesaleinternet.com>
<http://www.wholesaleinternet.com <http://www.wholesaleinternet.com>>
>  
>
>
>
> --
> Darin Steffl
> Minnesota WiFi
> www.mnwifi.com <http://www.mnwifi.com> <http://www.mnwifi.com/
<http://www.mnwifi.com/>>
> 507-634-WiFi
> Like us on Facebook <http://www.facebook.com/minnesotawifi
<http://www.facebook.com/minnesotawifi>>

-- 
====

Aaron

Re: 10g residential CPE

[Aaron Wendel]

Darin,

Our business support and residential support is the same department.  I 
have to pay those people to be in the office either way so it doesn't 
cost me any "more" to provide support for the residences. Yes, walking 
Grandma through getting her email can sometimes be a chore but that 
person is on the payroll whether he/she is helping Grandma or sitting 
there chatting with his/her co-worker.  If we dumped all the residential 
customers we would still have the same cost structure we do now.


Again, it's been free for the last 7 years at this point.  I've never 
been one to really do what I "should" anyway.


Aaron


On 12/28/2020 11:48 AM, Darin Steffl wrote:

Aaron,

The "Free" service doesn't cover your cost of support which is much 
higher for residential than any business customer. Our residential 
customers call at least 15x more often compared to business customers 
compared on a 1:1 ratio.


I honestly can't fathom providing free residential service because we 
make enough money on the business side of things. You should be 
charging something, at least $20-30 per month.


On Mon, Dec 28, 2020 at 11:15 AM Aaron Wendel 
mailto:aa...@wholesaleinternet.net>> wrote:


The $300 covers the equipment and the time to send someone out to a
house to install it.  If $300 is too much you can pay in 12
installments
of $25.

The TIK alone costs us about $250.

Aaron


On 12/27/2020 5:04 AM, Mark Tinka wrote:
>
>
> On 12/26/20 20:48, Darin Steffl wrote:
>
>> Aaron,
>>
>> One simple question. Why on earth would you offer free internet
>> service? How and why? Your site show 1 Gig symmetrical for free
when
>> you should be a minimum of $65 per month to be competitive.
>
> They also ask for no monthly fee after a single payment of US$300.
>
> Considering the 2Gbps package costs US$49.95, you'd guess they'd
value
> the 1Gbps service at, say US$27/month, give or take.
>
> So that US$300 provides a bit of coverage, perhaps 1 year, in which
> time they'd have likely upgraded the customer.
>
> Mark.

-- 


Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com <http://www.wholesaleinternet.com>




--
Darin Steffl
Minnesota WiFi
www.mnwifi.com <http://www.mnwifi.com/>
507-634-WiFi
Like us on Facebook <http://www.facebook.com/minnesotawifi>


--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com