Re: lots of internet starting at ~3 a.m. cst

[Paul Bradford]

You're right on and most of us don't monitor for shifts between direct
peering and transit very well.  we monitor for bw threshold.

Paul Bradford

Lead Network Engineer AS11776

C: 814-203-0699

E: pbradf...@breezeline.com



Breezeline.com

2875 Rt 764 Suite 2, Duncansville, PA 16635






On Tue, Jul 23, 2024 at 11:23 AM Mark Tinka  wrote:

>
>
> On 7/23/24 16:15, Bryan Holloway wrote:
>
> > What irks me is that we have direct PNIs with -- without naming names
> > -- the "big guys" delivering this content, and yet the majority of
> > this traffic is coming over our public IX connections and transit.
> >
> > Kinda defeats the purpose of a PNI ... anyone else seeing this?
>
> This issue is fairly common because more network is deployed in
> centralized locations that most people can access (like a large city
> data centre, e.t.c.), as well as for on-net caches. This is what leads
> to exchange point and PNI de-preference switching over to what an
> eyeball network may consider "transit".
>
> For the CDN and content folk, exchange point peering (whether PNI or via
> the exchange point fabric) is often under-spec'd by consequence, not by
> choice. So when clusters or the network feeding them suffers, they tend
> to be the first ones to be de-preferred for serving eyeballs. It happens
> a lot more often than you may realize, actually, and is not unique to
> any one CDN or content network.
>
> Mark.
>

Re: lots of internet starting at ~3 a.m. cst

[Paul Bradford]

Playstation 5 system updates kicked off around then:

https://www.reddit.com/r/PS5/comments/1ea1dii/ps5_firmware_update_released_for_july_23_patch/

On Tue, Jul 23, 2024 at 9:57 AM Aaron Gould  wrote:

> Thanks Jason, updates for what?  I was hoping any other eyeball network 
> operators may have been seeing "lots of internet" usage like me and may be 
> able to share what they know.  I'm always suspicious about the typical game 
> update that tends to cause something like this.  someone unicasted me a 
> response that it might be related to F1 Manager 2024 for video game consoles 
> released today
>
>
> i grabbed my netflow data for the last few hours for any source ip that has 
> sent more than 5 GB of data...
>
> Top 100 Src IP Addr ordered by bytes:
> Date first seen  Duration Proto   Src IP AddrFlows(%) 
> Packets(%)   Bytes(%) pps  bps   bpp
> 2024-07-23 04:37:29.600 14860.800 any  93.184.215.240   557558( 0.6)  
> 108.5 M( 9.2)  162.6 G(12.9) 7302   87.5 M  1497
> 2024-07-23 04:42:07.616 14577.664 any72.21.81.240   361240( 0.4)   
> 64.5 M( 5.5)   96.6 G( 7.7) 4426   53.0 M  1497
> 2024-07-23 06:52:08.192  6781.696 any 151.101.162.172   270298( 0.3)   
> 36.9 M( 3.1)   55.0 G( 4.4) 5445   64.9 M  1489
> 2024-07-23 04:23:08.160 15722.240 any 151.101.150.1721.0 M( 1.0)   
> 31.2 M( 2.6)   46.4 G( 3.7) 1985   23.6 M  1486
> 2024-07-23 04:42:47.552 14542.592 any  146.75.106.172   238287( 0.2)   
> 22.9 M( 1.9)   33.9 G( 2.7) 1575   18.7 M  1480
> 2024-07-23 03:40:11.776 18298.880 any 199.232.214.172   135793( 0.1)   
> 19.1 M( 1.6)   28.4 G( 2.3) 1043   12.4 M  1486
> 2024-07-23 03:48:59.392 17769.728 any 199.232.210.172   135811( 0.1)   
> 18.8 M( 1.6)   27.9 G( 2.2) 1057   12.6 M  1486
> 2024-07-23 04:22:05.184 15783.680 any  199.232.70.172   923350( 0.9)   
> 18.5 M( 1.6)   27.5 G( 2.2) 1169   14.0 M  1491
> 2024-07-23 04:36:40.704 14909.696 any   146.75.42.172   138813( 0.1)
> 5.6 M( 0.5)8.3 G( 0.7)  3744.5 M  1492
> 2024-07-22 22:50:26.048 35683.840 any  199.232.70.252   112323( 0.1)
> 5.4 M( 0.5)8.0 G( 0.6)  1501.8 M  1485
> 2024-07-23 06:52:06.656  6783.488 any   146.75.10.17274859( 0.1)
> 4.6 M( 0.4)6.8 G( 0.5)  6768.1 M  1489
> 2024-07-23 02:38:16.704 22011.392 any 151.101.160.204 6114( 0.0)
> 4.1 M( 0.3)6.2 G( 0.5)  1872.2 M  1489
> 2024-07-23 01:34:17.728 25851.136 any 151.101.161.19044001( 0.0)
> 4.1 M( 0.3)6.1 G( 0.5)  1571.9 M  1489
> 2024-07-22 22:17:50.464 37638.912 any 199.232.154.25269420( 0.1)
> 3.6 M( 0.3)5.4 G( 0.4)   961.1 M  1483
>
> -Aaron
>
>
>
> On 7/23/2024 8:19 AM, Peter Potvin wrote:
>
> Do you have *any* sort of additional information about this? Which source
> and destination ASNs, how much "a lot" is, etc? This sounds like a typical
> game update release cycle though without any information not a whole lot of
> networks would be able to confirm anything.
>
> Kind regards,
> Peter
>
>
> On Tue, Jul 23, 2024 at 9:07 AM Aaron Gould  wrote:
>
>> Anyone else see a lot of Internet traffic starting at 3 a.m. and
>> continuing even now?  Seems to be spiky tcp.
>>
>> --
>> -Aaron
>>
>> --
> -Aaron
>
>

-- 

Paul Bradford

Lead Network Engineer AS11776

C: 814-203-0699

E: pbradf...@breezeline.com



Breezeline.com

2875 Rt 764 Suite 2, Duncansville, PA 16635

Re: Out-of-Bailiwick DNS? (Was: HE.net problem)

[Paul Ebersman]

essen> I saw something online that said $250,000 but that didn't make
essen> sense if its all paperwork.

woody> Heh.  I see you are unfamiliar with ICANN.  They've said that
woody> same paperwork is likely to cost $375k in ICANN staff time for
woody> the next round.  Because, you know, inflation or something.

And $250k was not really the cost last time either. By the time you did
legal fees, deposits, fees, NOC, etc. it was closer to $500k per TLD.

If there is similar inflation on all the extra costs, that probably
means $600-700k next time.

I've been surprised that none of the folks that got TLDs seem to be
leveraging the technical/security brand protection like they could.

If I have an LG TV and it wants to update to .LG and LG is
DNSSEC signing the whole chain, that sure seems more likely to be legit
than .lg.tv or some such.

Add in that if you do brand, short of the root zone dumping you for some
reason, you own your experience and your full resolution chain. You can
pick who/how/where for all the labels, servers, anycast infra, etc.

But I haven't seen a lot of interest in taking advantage of that added
potential layer of security.

Re: Out-of-Bailiwick DNS? (Was: HE.net problem)

[Paul Ebersman]

ebersman> - don't have all your business critical domains under the same
ebersman>   registrar (unless it's of the CSC/markmonitor class)

jeroen> There is always going to be single point of failures in a
jeroen> hierarchical tree like that.

Everything in internet/infrastructure is risk tradeoffs and cost/benefit
analysis. If we could be perfect as engineers, we would be. ;)

Personally, the fact that the internet mostly functions most mornings
when I get up is still something that amazes me after years of using
it...

ebersman> - don't have all your auth NS for your domain in bailiwick
ebersman>   (within the domain being served)

jeroen> If, as it is the example in the thread, he.net 
jeroen> is your primary domain, which is their case, then if somebody in
jeroen> the tree disables the delegation of he.net ,
jeroen> nothing is going to fix resolution to you. Having your DNS
jeroen> servers in another TLD will not make he.net 
jeroen> appear in the global DNS again...

The above two points of mine tie together. If you can afford a registrar
who will be far more likely to care about you than random/bad
enforcement of external complaints and you're big/rich enough to be able
to use highly robust anycasted auth NS, in bailiwick is a much lower
risk.

If my "joe's fish shop and internet cafe" DNS is provided by "my mom let
me be a registrar if I ate my vegetables" diversity of TLD, registrar,
and auth NS (including out of bailiwick NS) becomes a much more
attractive and cheaper way to be more robust.

jeroen> Thus one only increases the risk by having multiple
jeroen> TLDs. Choosing a trusted registrar (one you have good contact
jeroen> with; indeed MM qualifies) and a TLD that will not cause you
jeroen> issues, is thus more important.

Again, this depends on scale. For SMB, multiple TLDs is more likely to
be a feature, for a really large business not so much. As Bill points
out, this is actually one of the few cases where brand TLD is a major
potential security upgrade.

Re: HE.net problem

[Paul Ebersman]

cjc> On the other side of this, we all may be learning the value of not
cjc> having all of you NS records in a single zone with a domain under a
cjc> single registrar.

>From some trainings I did on how to be sure your DNS was robust:

  - don't have all your business critical domains under the same
registrar (unless it's of the CSC/markmonitor class)
  - don't have all your auth NS for your domain in bailiwick (within the
domain being served)
  - don't have all your auth NS in the same routing domain (anycast can
be an exception to this if robust enough)
  - don't have the account registrar credential emails all within the
domain, nor with personal emails like gmail. do have them all under
control of your IT
  - protect all account credentials with strong passwords, MFA
  - have MX for your domain either with a very large provider or across
multiple domain names

It's painfully easy to fall off the internet and be unreachable if
you're not thinking about all this for business critical domains. You
don't ever want to be hoping that some customer kept your NOC phone
number in their phone. ;)

Re: HE.net problem

[Paul Ebersman]

jra> We have a report on outages that he.net has been placed in ICANN
jra> client hold, and people's DNS service is falling over on this
jra> Independence day.

Seems to have had hold removed 20:20 zulu, according to whois.

Domain back in .net and working again.

Re: Correcting national address databases?

[Christopher Paul via NANOG]

On 5/30/24 08:53, Mike Lewinski via NANOG wrote:

Another issue is that Amazon (and possibly other online retailers) are charging 
me and my neighbors excess sales tax based on the ZIP code associated with a 
town I do not live in. There's a way to complain and have it reversed for 
every single purchase.

I know this is out of our hands as network operators, but maybe some day one of 
you will be in a position to help.

I propose that there be a national LDAP service, with OUs for each 
zipcode (|ou=20500,dc=us,dc=gov)|. A household could register at 
USPS.gov and then be given write access to a household OU ("ou=1600 
Pennsylvania Ave NW,|ou=20500,dc=us,dc=gov|"). The household OU could 
then create inetOrgPersons under that, each of which would have 
self-write access.


--
Chris Paul | Rex Consulting |https://www.rexconsulting.net

Re: Fast backbone to NA from Asia

[Paul Rolland]

Hello,

On Wed, 22 May 2024 12:28:16 +0200
Mark Tinka  wrote:

> Asia-Pac <=> North America is typically faster via the Pacific, not the 
> Indian Ocean.
> 
> The Red Sea cuts would impact Asia-Pac <=> Europe traffic.

Yep, it hurts :(

 1. Gi0-3.rtr-01.PAR.witbe.net0.0%   1790.3   0.3   0.2  10.4   0.7
 2. 193.251.248.210.0%   1793.3   1.3   0.8  19.1   2.1
 3. bundle-ether305.partr2.saint-den  6.7%   179   87.1   4.5   1.1 156.7  17.4
 4. prs-b1-link.ip.twelve99.net  22.3%   1799.9  10.4   9.6  48.3   4.4
 5. prs-bb2-link.ip.twelve99.net  2.2%   179   10.4  10.3   9.8  27.3   1.3
 6. mei-b5-link.ip.twelve99.net   1.1%   178   17.3  18.1  17.2 115.1   7.4
 7. prs-bb1-link.ip.twelve99.net 27.5%   178  370.6 365.9 334.7 381.8   8.3
 8. ldn-bb1-link.ip.twelve99.net 68.5%   178  366.8 363.0 340.8 379.2   8.3
 9. nyk-bb2-link.ip.twelve99.net 11.8%   178  377.6 362.4 322.2 451.0  12.3
10. palo-b24-link.ip.twelve99.net50.8%   178  359.1 364.1 342.8 397.1   8.6
11. port-b3-link.ip.twelve99.net  0.0%   178  177.4 178.0 177.1 188.6   1.9
12. tky-b3-link.ip.twelve99.net  75.7%   178  355.2 364.0 339.8 377.5   8.0
13. tky-b2-link.ip.twelve99.net  50.0%   178  338.7 350.5 321.8 370.9  11.1
14. sng-b7-link.ip.twelve99.net  87.6%   178  307.8 318.6 306.8 332.0   6.7
15. sng-b5-link.ip.twelve99.net  86.4%   178  314.4 315.3 293.6 330.1  10.2
16. epsilon-ic-382489.ip.twelve99-cu 55.7%   177  364.8 362.9 346.5 391.8   9.1
17. 180.178.74.221   59.9%   177  357.6 366.9 343.4 562.7  25.8
18. swi-01-sin.noc.witbe.net 62.9%   176  374.7 366.4 346.6 381.3   8.3

1299 is now routing Paris to Singapore via US and Pacific... 
Not sure if transition 6 to 7 is what was expected, with a 350ms increase...

HE did/does that too, prefering to avoid any direct route from EU to Asia.

Paul

-- 
Paul RollandE-Mail : rol(at)witbe.net
CTO - Witbe.net SA  Tel. +33 (0)1 47 67 77 77
18 Rue d'Arras, Bat. A11Fax. +33 (0)1 47 67 77 99
F-92000 NanterreRIPE : PR12-RIPE

Please no HTML, I'm not a browser - Pas d'HTML, je ne suis pas un
navigateur "Some people dream of success... while others wake up and work
hard at it" 

"I worry about my child and the Internet all the time, even though she's
too young to have logged on yet. Here's what I worry about. I worry that 10
or 15 years from now, she will come to me and say 'Daddy, where were you
when they took freedom of the press away from the Internet?'"
--Mike Godwin, Electronic Frontier Foundation 


pgpDOQ3cp3Vtj.pgp
Description: OpenPGP digital signature

Re: Netskrt - ISP-colo CDN

[Paul Bradford]

I have some on my network.  I don't think they populate content from their
own cdn network, but it comes from Amazon.   interestingly for the NFL
super bowl, while paramount+ streamed the game, on Amazon Prime Video you
could "Watch super bowl on paramount+ Via Prime.".  that did actually drive
users to using the netskrt caches.

They seem to work OK.  TNF in 6 months will tell us more.  :)



On Thu, Apr 4, 2024 at 6:14 PM John Stitt  wrote:

> The website says they are part of the Streaming Video Technology Alliance.
>
>
>
> I wonder if this is a prepackaged Open Cache box.
>
>
>
> https://opencaching.svta.org/
>
>
>
> We also don’t appear to have had any traffic from them.  Not much on the
> peeringdb for the USA ASN either.
>
>
>
> BGP.tools shows they have upstreams with each ASN, and are on Ohio IX with
> AS53471, but not really any peers anywhere.  Looks like Cogent and Zayo for
> upstreams and only peer I see is AS1239 (Sprint Wireline (Cogent))
>
>
>
> John Stitt
>
>
>
> *From:* NANOG  *On
> Behalf Of *Aaron Gould
> *Sent:* Thursday, April 4, 2024 4:36 PM
> *To:* Eric Dugas 
> *Cc:* nanog@nanog.org
> *Subject:* Re: Netskrt - ISP-colo CDN
>
>
>
> You don't often get email from aar...@gvtc.com. Learn why this is
> important 
>
> Thanks... they told me it was free.
>
> -Aaron
>
> On 4/4/2024 4:12 PM, Eric Dugas wrote:
>
> That name rang a bell so I looked up my emails.
>
>
>
> They contacted me last year, they were claiming to be "working with some
> of the major streaming brands, such as Amazon Prime Video, to improve the
> quality of both VOD and live streaming while also reducing the load on ISP
> networks such as your own.".
>
>
>
> Based on my quick research, they have a few registered ASNs (their peeringdb
> page ) with a few netblocks but I
> get 0 traffic from them (we're a sizable eyeball network). Their origin
> network might still not be ready but digging a little bit more, it seems
> they act as a third-party video caching solution and not as an origin CDN
> so in the end, they're really just trying to sell ISPs and other types of
> customers their caching solutions.
>
>
> Eric
>
>
>
> On Thu, Apr 4, 2024 at 4:00 PM Aaron Gould  wrote:
>
> Anyone out there using Netskrt CDN?  I mean, installed in your network
> for content delivery to your customers.  I understand Netskrt provides
> caching for some well known online video streaming services... just
> wondering if there are any network operators that have worked with
> Netskrt and deployed their caching servers in your networks and what
> have you thought about it?  What Internet uplink savings are you seeing?
>
> Netskrt - https://www.netskrt.io/
>
>
> --
> -Aaron
>
> --
>
> -Aaron
>
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe. If you are not expecting this message contact the
> sender directly via phone/text to verify.
>
>
>

Re: N91 Women mixer on Sunday?

[Paul WALL]

Hi, Anne-

I'm sure that your time was better spent gathering the "credentials"
in your signature, but I checked the last 20 or so NANOG meetings and
didn't see a single registration from you, so perhaps stay out of
things you know literally nothing about.

If it weren't for Ilissa, NANOG would not exist in the form that it
does today, and she's done more work on and off the clock driving the
success of the organization and their meetings than she takes credit
for. NANOG, and especially the women that attend NANOG, owe her a
tremendous debt of gratitude. Her opinion, and Tina's response, are
literally the only ones that carry any weight in this thread, period.

--
Drive Slow,

Paul Wall
Rapper, Retired, and Actor
Swishahouse Alum
Author: Get Money, Stay True
Nominated: Best Rap Performance as a Duo or Group
Winner: Best Rap Collaboration
Winner: Best Rap/R Collaboration
Winner: Taste Maker (Style and Trendsetter)
Contributor: Midnight Club 3: DUB Edition for Xbox and PlayStation 2 –
"Sittin' Sidewayz"
Author: The Peoples Champ
Emeritus: Houston University (no degree)


On Thu, Mar 28, 2024 at 3:24 PM Anne P. Mitchell, Esq.
 wrote:
>
>
>
> > I'm not sure people realize how much crap that staff and the PC get *every 
> > meeting* about the agenda. There's always someone unhappy because this 
> > event wasn't the same, or why was it in this room over here, or OMG Wed 
> > afternoon, etc. Having seen how that sausage gets made, they don't get 
> > enough credit.
>
> Having been the chair of the Asilomar Microcomputer workshop, and the founder 
> and chair of the original Email Deliverability Summits, as well as organizing 
> many legal conferences, I have to say "^^^ this, 1000%."
>
> Furthermore:
>
> > On Mar 28, 2024, at 11:45 AM, Ilissa Miller  wrote:
> >
> > For those that know me, I rarely provide constructive input about NANOG 
> > matters
>
> ..and you haven't here, either.  Pointing fingers and griping about things is 
> not constructive.  If you really care about this issue, then get involved and 
> help change it.
>
> Anne
>
> ---
> Anne P. Mitchell, Esq.
> Internet Law & Policy Attorney
> CEO Institute for Social Internet Public Policy (ISIPP)
> Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal email marketing 
> law)
> Board of Directors, Denver Internet Exchange
> Dean Emeritus, Cyberlaw & Cybersecurity, Lincoln Law School
> Prof. Emeritus, Lincoln Law School
> Chair Emeritus, Asilomar Microcomputer Workshop
> Counsel Emeritus, eMail Abuse Prevention System (MAPS)
>
>
>
>
>

On Thu, Mar 28, 2024 at 3:24 PM Anne P. Mitchell, Esq.
 wrote:
>
>
>
> > I'm not sure people realize how much crap that staff and the PC get *every 
> > meeting* about the agenda. There's always someone unhappy because this 
> > event wasn't the same, or why was it in this room over here, or OMG Wed 
> > afternoon, etc. Having seen how that sausage gets made, they don't get 
> > enough credit.
>
> Having been the chair of the Asilomar Microcomputer workshop, and the founder 
> and chair of the original Email Deliverability Summits, as well as organizing 
> many legal conferences, I have to say "^^^ this, 1000%."
>
> Furthermore:
>
> > On Mar 28, 2024, at 11:45 AM, Ilissa Miller  wrote:
> >
> > For those that know me, I rarely provide constructive input about NANOG 
> > matters
>
> ..and you haven't here, either.  Pointing fingers and griping about things is 
> not constructive.  If you really care about this issue, then get involved and 
> help change it.
>
> Anne
>
> ---
> Anne P. Mitchell, Esq.
> Internet Law & Policy Attorney
> CEO Institute for Social Internet Public Policy (ISIPP)
> Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal email marketing 
> law)
> Board of Directors, Denver Internet Exchange
> Dean Emeritus, Cyberlaw & Cybersecurity, Lincoln Law School
> Prof. Emeritus, Lincoln Law School
> Chair Emeritus, Asilomar Microcomputer Workshop
> Counsel Emeritus, eMail Abuse Prevention System (MAPS)
>
>
>
>
>

Re: Any info on AT Wireless Outage?

[Paul Bradford]

missing an   permit ip any any?  classic



On Wed, Feb 28, 2024 at 10:56 AM  wrote:

> I read it as “someone pushed an ACL that wasn’t properly reviewed and it
> really screwed things up."
>
> On Feb 27, 2024, at 21:41, Mark Seiden  wrote:
>
> aside from the official pablum that was released about an “incorrect
> process used”
> (which says exactly nothing) does anyone actually know anything accurate
> and
> more specific about the root cause?
>
> (and why it took 11 hours to recover?)
>
> On Feb 22, 2024, at 11:15 AM, John Councilman 
> wrote:
>
> From what I've read, they lost their database of SIM cards.  I could be
> wrong of course.
>
> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:
>
>> As widespread as it seemed to be, it feels like it would be quite a trick
>> if it were a single piece of hardware.  Firmware load that ended badly, I
>> wonder?
>>
>>
>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
>> wrote:
>>
>>> Do you have the ability to expand on this at all? Do you mean a hardware
>>> failure of some kind IE router, optitcs, etc?
>>>
>>>
>>>
>>> *From:* NANOG  *On
>>> Behalf Of *R. Leigh Hennig
>>> *Sent:* Thursday, February 22, 2024 8:17 AM
>>> *To:* Robert DeVita 
>>> *Cc:* nanog@nanog.org
>>> *Subject:* Re: Any info on AT Wireless Outage?
>>>
>>>
>>>
>>> Word around the campfire is that it’s a Cisco issue.
>>>
>>>
>>>
>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>>> wrote:
>>>
>>>
>>>
>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Robert DeVita**​**​**​**​*
>>>
>>> *CEO and Founder*
>>>
>>> t: (469) 581-2160 <(469)%20581-2160>
>>>
>>>  |
>>>
>>> m: (469) 441-8864 <(469)%20441-8864>
>>>
>>> e: radev...@mejeticks.com
>>>
>>>  |
>>>
>>> w: mejeticks.com 
>>>
>>> a:
>>>
>>> 2323 N Akard Street
>>>
>>> ,
>>>
>>> Dallas
>>>
>>> ,
>>>
>>> 75201
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>>
>>>
>>>
>>> The risk of trading futures and options can be substantial. All
>>> information, publications, and material used and distributed by Advance
>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>>> independent research department as defined in CFTC Regulation 1.71.
>>> Information obtained from third-party sources is believed to be reliable,
>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>>> is not necessarily indicative of future results.
>>>
>>
>
>

Re: NANOG 90 Attendance?

[Paul Ebersman]

mhammett> This seems more ideological and not overly appropriate for
mhammett> NANOG.

No, covid protocols are something that every conference that is serious
about inclusion should be *very* concerned with.

Saying that NANOG doesn't care about this says that NANOG can't be
bothered to make an effort to make the conference safe for more folks.

There's a reason I'm not there in person, even though I've attended for
years, spoken there, and volunteered for multiple rounds on committees.

Re: BGP Engines with support to "RTFilter address-family"

[Paul Rolland]

Hello,

On Sun, 26 Feb 2023 17:46:42 -0300
Douglas Fischer  wrote:

> But I'm looking for an open-source engine that supports it.
> 
> The official FRR documentation does not mention anything about RFC 4364,
> or RTFilter address family.
> So, I think FRR does not support RTFilter Constrained Route Distribution.
> 
> Do any of the colleagues have any suggestions on this?

ExaBGP ?

https://github.com/Exa-Networks/exabgp/wiki/RFC-Information

Best,
Paul

-- 
Paul RollandE-Mail : rol(at)witbe.net
CTO - Witbe.net SA  Tel. +33 (0)1 47 67 77 77
18 Rue d'Arras, Bat. A11Fax. +33 (0)1 47 67 77 99
F-92000 NanterreRIPE : PR12-RIPE

Please no HTML, I'm not a browser - Pas d'HTML, je ne suis pas un
navigateur "Some people dream of success... while others wake up and work
hard at it" 

"I worry about my child and the Internet all the time, even though she's
too young to have logged on yet. Here's what I worry about. I worry that 10
or 15 years from now, she will come to me and say 'Daddy, where were you
when they took freedom of the press away from the Internet?'"
--Mike Godwin, Electronic Frontier Foundation 


pgpp3q0eldPtB.pgp
Description: OpenPGP digital signature

Re: ROA Will Expire Soon - ARIN

[Paul Emmons]

In our experience, I think, we do a 24 month rpki cert tied the key shared
with ARIN. You simply create a new rpki cert in the ARIN hosted service.
Due operational reasons we will delete an old cert a month after publishing
the new cert just to keep things clean.  We don't have a lot of space
turnover so we will typically do a new cert 2 or 3 times a year.

If your underlying resources are pretty much static, just make your cert
good for as long as you can.

On Fri, Sep 9, 2022, 9:08 AM Ca By  wrote:

>
>
> On Fri, Sep 9, 2022 at 9:04 AM Brad Gorman  wrote:
>
>> A message is sent to points of contact of an Org one month before
>> expiration of a ROA in the ARIN repository.  At any time prior to the ROA
>> expiry, a new (duplicate) ROA can be created for the same resources with a
>> new expiry date in the future. The soon to expire ROA can be deleted once
>> the new ROA has been published to the repository or you can simply wait for
>> it to expire.
>>
>>
>>
>>
>>
>> Brad
>>
>>
> Any chance arin can post a step by step guide on the arin website?
>
> Seems like a big deal to have an roa expire, and a well documented process
> will create a lot of confidence.
>
> As where an expired roa outage will cause a company to never use rpki
> again.
>
>>
>>
>> *From: *NANOG  on behalf of Ca
>> By 
>> *Date: *Friday, September 9, 2022 at 10:12 AM
>> *To: *John Sweeting 
>> *Cc: *North American Network Operators' Group 
>> *Subject: *Re: ROA Will Expire Soon - ARIN
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 9, 2022 at 5:21 AM John Sweeting  wrote:
>>
>> You can contact the ARIN Helpdesk at +1-703-227-0660. Someone will also
>> be sending you an email off list.
>>
>>
>>
>> John
>>
>>
>>
>> Where is ARIN’s documented procedure for how hosted ROAs handle renewal
>> prior to expiration ?
>>
>>
>>
>>
>>
>>
>> Sent from my iPhone
>>
>> > On Sep 9, 2022, at 8:01 AM, Terrance Devor  wrote:
>> >
>> >
>> > Can someone from ARIN please reach out to me. We don't want the ROA to
>> expire...
>> >
>> > Kind Regards,
>> > Terrance
>>
>>

RE: Router ID on IPv6-Only

[Paul Amaral via NANOG]

Is there really such as thing as pure IPV6 only? I don’t think you will be able 
to run IPV6 for transport without the router locally knowing how to handle 
IPV4, at least not right now as there’s a lot of legacy code. Usually IPV6 is 
enabled longer after IPV4 has been running. With that said, can’t you just 
enable ipv4 and not route it passed the router, then use RFC1918 to manually 
general your 32 bit ID.

 

Paul 

 

From: NANOG  On Behalf Of Crist Clark
Sent: Thursday, September 8, 2022 1:39 AM
To: nanog@nanog.org list 
Subject: Router ID on IPv6-Only

 

During some IPv6 numbering discussions at work today, someone had a question 
that I hadn't really considered before. How to choose 32-bit router IDs for 
IPv6-only routers.

 

Quick background. We have a requirement to convert a significant portion of our 
network to IPv6-only over the next few years. Previously, I, and everyone else 
on the team, have only ever set up routers in dual-stack environments. Choosing 
a router ID for use in routing protocols just followed whatever rules you used 
for your IPv4 networking. You used the same router ID in IPv4 and IPv6.

 

Well, now there is no IPv4. But BGP, OSPFv3, and other routing protocols still 
use 32-bit router IDs for IPv6. On the one hand, there are plenty of 32-bit 
numbers to use. Generally speaking, router IDs just need to be unique inside of 
an AS to do their job, but (a) for humans or automation to generate them and 
(b) to easily recognize them, it's convenient to have some algorithm or 
methodology for assigning them.

 

Has anyone thought about this or have a good way to do it? We had ideas like 
use bits 32-63 from an interface. Seems like it could work, but also could 
totally break down if we're using >64-bit prefixes for things like 
router-to-router links or pulling router loopbacks out of a common /64.

 

Also, various network OS implementations will typically automatically choose a 
router ID from the IPv4 addresses on the router by some algorithm (e.g. 
numerically lowest) if not explicitly configured. Was curious what IPv6-only 
routers do. Haven't had the chance to get on some lab gear or GNS3 to just try 
it and see.

Re: cogent - Sales practices

[Paul Emmons]

Two current experiences . . .
I still do work with an ILEC that gets requests for waves to Cogent. Cogent
has a data center in the market but won't allow the ILEC to build in.  So
Cogent burns ports in another data center where Cogent pays for space and
power.  Cogent reps says no one gets anything for free.

Had a recent event with Meta for the small market ixp deployment.  The use,
just use, Cogent for connectivity /fill.

And a plan . . .  Order up 10g Global peer exchange ports from them. They
are free.  Maybe if they sell enough of those they will have to spend more
of their dollars.

Re: Akamai Peering

[Paul Emmons]

Akamai isn't supporting 10g ports on IXPs.  I'd be surprised if the allowed
it on PNIs.  As for not being on the IXPs, that's odd.

On Tue, Jul 26, 2022 at 8:23 AM Jawaid Bazyar 
wrote:

> Hi,
>
>
>
> We had Akamai servers in our data center for many years until a couple
> years ago, when they said they’d changed their policies and decommissioned
> the servers.
>
>
>
> I understand that, maintaining many server sites and being responsible for
> that hardware, even if you pay nothing for power or collocation, must be
> costly. And at the time, we didn’t have much traffic to them.
>
>
>
> Today, however, we’re hitting 6 Gbps with them nightly. Not sure what
> traffic it is they’re hosting but it’s surely video of some sort.
>
>
>
> We are in the same data center with them, Edgeconnex Denver, and they
> refuse to peer because they say their minimum traffic level for peering is
> 30 Gbps.
>
>
>
> Their peeringdb entry says “open peering”, and in my book that’s not open
> peering.
>
>
>
> So this seems to be exactly backward from where every other major content
> provider is going – free peering with as many eyeball networks as possible.
>
>
>
> Google – no bandwidth minimum, and, they cover costs on 1st and every
> other cross connect
>
> Amazon – peers are two Denver IX
>
> Apple – peers at two Denver IX
>
> Netflix – free peering everywhere
>
>
>
> And, on top of that, Akamai is not at either of the two Denver exchange
> points, which push together probably half a terabit of traffic.
>
>
>
> What is the financial model for Akamai to restrict peering this way?
> Surely it’s not the 10G ports and optics, which are cheap as dirt these
> days.
>
>
>
> Doesn’t this policy encourage eyeballs to move this traffic to their
> cheapest possible transit links, with a potential degradation of service
> for Akamai’s content customers?
>
>
>
> Thanks for the insight,
>
>
>
> Jawaid
>
>
>
>
>
> *[image: uc%3fid=1CZG_hGEeUP_KD95fSHu2oBRA_6dkOo6n]*
>
> *Jawaid Bazyar*
>
> Chief Technical Officer
>
> VERO Broadband
>
> [image: signature_3735065359]
>
> 303-815-1814
>
> [image: signature_3363732610]
>
> jbaz...@verobroadband.com
>
> [image: signature_60923]
>
> https://verobroadband.com
>
> [image: signature_4057438942]
>
> 2347 Curtis St, Denver, CO 80205
>
>
>

Re: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

[Paul Rolland]

Hello,

On Thu, 21 Jul 2022 12:39:55 -0400
Tom Beecher  wrote:

> Well that shows my assertion was probably wrong.
> 
> Given the geopolitical situation between the US and China, along with
> certain government orders, could likely infer this is intentional.

Well, I remember VZN when it was UUNet... and then, they had 3 main ASNs:
 - 701 (US)
 - 702 (EU)
 - 703 (APAC)

Playing with the LG, it may seem that the route is visible in the "703
region", so that may be traffic engineering, geo-whatever reason, config
mistake, ...

Paul

-- 
Paul RollandE-Mail : rol(at)witbe.net
CTO - Witbe.net SA  Tel. +33 (0)1 47 67 77 77
18 Rue d'Arras, Bat. A11Fax. +33 (0)1 47 67 77 99
F-92000 NanterreRIPE : PR12-RIPE

Please no HTML, I'm not a browser - Pas d'HTML, je ne suis pas un
navigateur "Some people dream of success... while others wake up and work
hard at it" 

"I worry about my child and the Internet all the time, even though she's
too young to have logged on yet. Here's what I worry about. I worry that 10
or 15 years from now, she will come to me and say 'Daddy, where were you
when they took freedom of the press away from the Internet?'"
--Mike Godwin, Electronic Frontier Foundation 


pgpOayVJpHCKu.pgp
Description: OpenPGP digital signature

Re: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

[Paul Rolland]

Hello,

On Thu, 21 Jul 2022 12:20:37 -0400 (EDT)
Jon Lewis  wrote:

> I looked at this a little last night, but didn't have time to write an 
> email about it.  Verizon has a lookingglass:
> 
> https://www.verizon.com/business/why-verizon/looking-glass/
> 
> which you can use to see that Verizon has no route covering 182.61.200.0. 
> Looking at routeviews, I see routes for 182.61.200.0/22, 
> and 182.61.200.0/21, but no path via Verizon.

Just tested using:
  Region: Asia Pacific
  Location: HKG
  Show BGP route
  Net: 182.61.200.0

and the answer is:

182.61.200.0/22 (2 entries, 1 announced)
*BGPPreference: 170/-101
Age: 4w0d 22:32:07  Metric: 10  Metric2: 500502 
Announcement bits (4): 0-KRT 3-RT 8-BGP_RT_Background 9-Resolve 
tree 4 
AS path: (65336) 4134 23724 38365 I  (Atomic Originator)
Communities:   
Localpref: 100

Paul


pgpblDby5RqkH.pgp
Description: OpenPGP digital signature

Re: Frontier Dark Fiber

[Paul Timmins]

Your rights under the ICA are dead. Since 2002 you were only able to 
order it if one end was in a tier 3 wirecenter, and it was killed in 
2021 as an orderable product.



https://www.federalregister.gov/documents/2021/01/08/2020-25254/modernizing-unbundling-and-resale-requirements-in-an-era-of-next-generation-networks-and-services


There's an 8 year transition for existing unbundled dark fiber (February 
28, 2029). Dark fiber loops were dead in 2002 under the TRRO.





On 7/13/22 07:45, Mike Hammett wrote:

Oh, and I forgot to mention that my ICA has it.



-
Mike Hammett
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 


*From: *"Mike Hammett" 
*To: *nanog@nanog.org
*Sent: *Wednesday, July 13, 2022 6:40:47 AM
*Subject: *Frontier Dark Fiber

I'm looking for a contact at Frontier that can discuss dark fiber.

My current account exec says they don't offer it, yet prior 
conversations with him and a previous SE revealed that they very much 
did (just didn't have availability on the paths I wanted at the time).


Their web site highlights it fairly proudly.


I'm aware that availability varies.

I'm aware that they likely don't want to sell it.



-
Mike Hammett
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Paul Timmins]

How many times have I seen an installer only download the parts it needs 
vs just reinstall the next version right over top of the existing 
version? I know stuff like xplane seems to do a comparison of file 
signatures and only downloads the changed parts for the updates between 
whatever version I have and whatever version is current now, but I'd 
imagine a lot of installers these days just take advantage of the fact 
the user has a super fast connection and they don't have to care about 
shipping the entire new installer just to run an update.


Not to mention whatever amounts of shovelware come with a few megabyte 
print driver for a modern printer/scanner/copier. Let's just include a 
copy of McAfee endpoint protection in this java update in case the user 
opts into selecting that as an option during install? etc.


-Paul

On 6/6/22 14:24, Chris Adams wrote:

Once upon a time, Michael Thomas  said:

I meant downloads as in gigantic games. If you give them more
bandwidth it just encourages the game makes to build bigger game
downloads.

I don't buy that - users are still constrained on storage, especially on
consoles.

RE: Any sign of supply chain returning to normal?

[Paul Amaral via NANOG]

We have customers being forced to use EOL products that they previously 
replaces as they continue to wait on the vendor for new EQ.
Paul

-Original Message-
From: NANOG  On Behalf Of Dave Taht
Sent: Thursday, May 19, 2022 12:16 PM
To: Jason Biel 
Cc: NANOG 
Subject: Re: Any sign of supply chain returning to normal?

On Thu, May 19, 2022 at 11:01 AM Jason Biel  wrote:
>
> > The oem ain't gonna support the resold device either.
>
> Many vendors support resold gear through a recertification cost in order to 
> bring it back under a support contract.

In my world, support ends after 6 months. Period.

It's even worse than that. Mediatek, for example, provides a devkit to new 
customers, still, based on the obsolete
LEDE-17 release of openwrt, e.g. 6+ year code. I recently pointed out to a 
marketing manager pimping how wifi-7 was going to fix latency on wifi in 3-4 
years, how crappy the factory driver was, compared to what's now in linux

https://blog.cerowrt.org/post/fq-codel-unifi6/

and asked when they were going to ship that instead, to a blank stare... And 
yes, I know of several "new" customers for that chipset that are using that 
obsolete code, too scared and incompentent to make the jump to a more current 
OS.

If you think that's bad, qualcomm is worse, and I just established a new 
record, I think, with truly ancient broadcom's openwrt based devkit that just 
shipped with the "NEW" triband tp-link deco series...
https://www.tp-link.com/us/deco-mesh-wifi/product-family/deco-xe75/ - I can 
hardly bring myself to talk to the sea of CVEs and incompetence in there... you 
can start with them STILL shipping dnsmasq 2.62... and linux 3.3.8.

I used to be really proud that openwrt was used by all these major 
manufacturers, but I'd also thought that they'd have been responsible enough to 
at least keep up with CVEs, and stay within a few years of the mainline.

If you are wondering why WiFi-6 works so badly out of the box, or why
ipv6 is not rolling out, you don't have to look much further. The really, 
really sad thing, is that the ODM in these cases, just slaps the devkit and a 
fancy gui on top of it, and ships the product, with no further support.

So I kind of view recycling routers, with newer software, as a great way to 
clean up the present ecosystem. And if you  looked at the first url I pasted 
above, with 4x more throughput, and 10x less latency, on "obsolete", hw.

> On Thu, May 19, 2022 at 10:58 AM Dave Taht  wrote:
>>
>> On Thu, May 19, 2022 at 10:33 AM Jason Biel  wrote:
>> >
>> > Who's going to support that reflashed device? Certainly not the OEM vendor.
>>
>> The oem ain't gonna support the resold device either.
>>
>> Yes, arguably, someone or someones doing a value add would have to be 
>> making money at it somehow.
>>
>> However, at least in my world, volunteers make the world round, still.
>> It would kind of suck,
>> I suppose, if someone unleashed a few hundred thousand reflashed 
>> routers like the TIP openwifi effort ( 
>> https://telecominfraproject.com/ ) seem to intent on doing...
>>
>> ... but if the OS is good enough to not need support, the impact is minimal.
>> >
>> > On Thu, May 19, 2022 at 10:30 AM Dave Taht  wrote:
>> >>
>> >> On Thu, May 19, 2022 at 9:07 AM NetEquity Sales  
>> >> wrote:
>> >> >
>> >> > As someone who works within the "secondary market" for networking 
>> >> > hardware, there is a ton of demand spilling over into the 
>> >> > "pre-owned/vendor refurbished" market.
>> >>
>> >> I just wish there were people putting in a value-add, like 
>> >> reflashing with better software, first.
>> >>
>> >> >
>> >> > Market prices on pre-owned equipment are rapidly increasing in step 
>> >> > with increased demand and dwindling supply.
>> >> >
>> >> > Market prices on 1G - 10G switching products, wireless infrastructure 
>> >> > devices, etc have been rising precipitously. Even semi "legacy" stuff 
>> >> > going back 2-3 generations (EOL/EOS) from current gen have doubled, 
>> >> > tripled, even quadrupled in price.
>> >> >
>> >> > I've been involved in the hardware business for 20 years and the 
>> >> > current market landscape is unprecedented.
>> >> >
>> >> >
>> >> >
>> >> > Cory J. Andrews
>> >> > ++
>> >> > NetEquity.com (Formerly CiscoBuy.com)
>> >> > 4519 Northgate Court
>> >> > Sarasota, FL 34234
>&g

Re: Disney+ Issues

[Paul Thornton]

On 29/04/2022 18:21, Norman Jester wrote:


We're having a heck of a time with this, customers are posting all
over social media about it etc.
The company who does their ip classification is Neustar and we have
been talking to them.
In our case, it is a /17 that moved from Germany to us in the UK after a 
purchase a year ago.


If we look up the block in question on Neustar's website, it correctly 
geolocates to the UK - and they correctly fetch the ISP information from 
RIPE.  This has been the case for ages (and I don't think Neustar has 
ever shown it incorrect after we notified them - along with other geoloc 
companies - that it was now UK-based).


So if Disney+ are using Neustar, they are caching the results somehow or 
applying their own secret sauce that gets it wrong.


Paul.

Re: Disney+ Issues

[Paul Thornton]

On 29/04/2022 14:22, Josh Luthman wrote:


Did you try:

Disney+: E-mail them the trouble subnet at 
techops-distribut...@disneystreaming.com. Also, 
techops-servi...@disneystreaming.com will probably be where that sends 
you. Another possible email is disneyplusispsupp...@disneyplus.com.


https://thebrotherswisp.com/index.php/geo-and-vpn/



We too are having the same issue - started suddenly around 6-8 weeks ago 
having worked fine for at least a year.  I have no idea what they 
changed.  Based on my first hand knowledge, these E-mail addresses go 
nowhere where anyone either can - or wants to - resolve issues.


Disney+ appear to be the worst outfit at handling this kind of thing: 
They have no concept of a service provider wanting them to update an 
entire block - they are fixing this for individual customers who call 
them but we are calling them weekly, and E-mailing regularly too; but go 
around in circles where someone promises to call back having sorted it.  
This never happens.


They also appear to use some opaque geoloc service (who themselves don't 
have a "you have this wrong" button) and really don't care that they are 
making life difficult for their paying customers!


We have to keep telling new customers variations of "Yes, this is 
Disney's fault, no we can't fix it" which doesn't go down very well 
because "It worked fine with my previous provider, it must be your 
issue".  Apart from suggesting they cancel their subscription because of 
Disney's incompetence there's not much else we can do :(



I get that you have to appease rights holders and do this idiotic 
geolocation thing, because they are still obsessed with geographical 
boundaries in the 21st century.  But if you are going to do this, can 
you please damned well fix *your* screwups when you get it wrong in a 
timely manner - or don't bother doing it at all.



Paul.

Re: Sabotage: several severed cables at the origin of a major internet outage in France

[Paul Ferguson]

On 4/27/22 7:08 AM, Sean Donelan wrote:



Multiple physical cable cuts in multiple diverse locations in France.

Several networks that connect the internet infrastructures of major 
French cities were cut overnight, in a short interval. A state source 
evokes with "the Obs" a "coordinated malicious act", which confirms SFR 
and Free affected. An investigation has been opened.


https://www.nouvelobs.com/faits-divers/20220427.OBS57722/plusieurs-cables-sectionnes-a-l-origine-d-une-importante-panne-internet-en-france.html 





English language news article, fwiw:

https://www.telegraph.co.uk/world-news/2022/04/27/internet-multiple-cities-across-france-suspected-sabotage/

Cheers,

- ferg


--
Paul Ferguson
Tacoma, WA  USA
Illegitimi non carborundum.

Re: Cogent ...

[Paul Timmins]

On 3/31/22 11:38, Laura Smith via NANOG wrote:

However, perhaps someone would care to elaborate (either on or off-list) what 
the deal is with the requirement to sign NDAs with Cogent before they'll 
discuss things like why they still charge for BGP, or indeed any other 
technical or pricing matters. Seems weird ?!?


Same reason your employer doesn't want employees telling each other 
their salary. Not every similarly situated customer pays the same for 
the same service.

Re: Let's Focus on Moving Forward Re: V6 still not supported

[Paul Rolland]

Hello,

On Sat, 26 Mar 2022 09:35:30 -0400
"Abraham Y. Chen"  wrote:

> touching the hardware, by implementing the EzIP technique (*/disabling/* 
> the program code that has been */disabling/* the use of the 240/4 
> netblock), an existing CG-NAT module becomes a RAN! As to universal 

Have you ever considered that this may be in fact:

*/writing/* and */deploying/* the code that will allow the use of 240/4 the
way you expect 


Paul


pgp6kGDmOvUU6.pgp
Description: OpenPGP digital signature

Re: "Permanent" DST

[Paul Ebersman]

eric> If Canada doesn't do the same thing at the same time, it'll be a
eric> real hassle, dealing with a change from -8 to -7 crossing the
eric> border between BC and WA, for instance. It has to be done
eric> consistently throughout North America.

You must not have ever dealt with Indiana, where it was DST or not by
choice per county. It wasn't quite the cluster***k you'd think.

Re: LEC copper removal from commercial properties

[Paul Emmons]

Saw this

https://www.nojitter.com/consultant-perspectives/decommissioning-copper-gets-real

Re: LEC copper removal from commercial properties

[Paul Emmons]

Do MSOs and CLEC/fiber providers require free power and space?

On Wed, Feb 16, 2022, 7:59 PM Martin Hannigan  wrote:

>
> NANOG'ers;
>
> At least in Boston, commercial property owners are receiving notices that
> 'copper  lines are being removed per FCC rules' and replaced with fiber.
> The property owner, not the network operators (or users of unbundled
> elements if that's even still a thing) are being presented with an
> agreement that acknowledges the removal, authorizes the fiber installation
> and provides for a minor oversight of the design. It suggests that no costs
> are involved in terms of hosting equipment. No power reimbursement. No rent
> for spaces used.
>
> There is an ominous paragraph in the letter that says if the property
> owner doesn't comply that tenants will lose all services including elevator
> phones, alarms, voice, internet and any copper/ds0 originated services.
> They didn't say 911, but that would go without saying.
>
> Has anyone heard of this?
> What FCC rule requires this?
>
> Thanks for any insights.
>
> Warm regards,
>
> Martin
>

Re: 25G SFP28 capable of rate-adaption down to 1G?

[Paul Emmons]

We have done that with a CVR and 1g sfp.

On 1/31/2022 11:05 AM, Bill Woodcock wrote:

Hey, does anyone know of an SFP28 capable of rate-adapting down from 25G on the 
cage side down to 1G on the line side?  Can be copper or fiber on the line 
side, I don’t care, my interest is in the chip inside.

Thanks,

 -Bill

Re: Long hops on international paths

[PAUL R BARFORD]

Hello Adam,

Thanks for your follow-up - it helps to clarify that we've been observing in 
our analysis.  The tradeoff between cost (100G+ links), manageability (related 
to BGP) and risk (i.e., concentration of high-value links in a few locations) 
is really interesting and not something that we were aware of.

PB

From: athomp...@merlin.mb.ca 
Sent: Tuesday, January 25, 2022 10:54 AM
To: PAUL R BARFORD ; davidbass...@gmail.com 

Cc: nanog@nanog.org 
Subject: RE: Long hops on international paths


Peering connection, I think, can explain this.

With some notable exceptions (all of whom participate here, I think), carriers 
still don’t throw around 100G+ peering routers around like sprinkles on a 
donut.  (Even those big networks don’t, it just looks like they do because 
they’re freaking huge.)

I suspect what you may be seeing is NOT “international carriers all concentrate 
on a single router” at all, but rather a) “large carriers tend to interconnect 
at only a few points” and b) “the best path from North America to Europe will 
therefore tend to always go through the same small set of inter-AS peering 
routers on this side”.

What you’ve described, purely in the public-visible layer 3 internet, is normal 
in my experience.  I would fully expect upwards of 90% of my daily traffic 
crosses one of 3 peering routers in my network, no surprise there, but ALSO I 
estimate at least 80% of it crosses one of no more than 5 or 6 routers even as 
I get 5, 6, 7, 8 hops away from me.  The internet isn’t as diversely-pathed as 
it once was.

In the MPLS carrier case, I’m aware of a couple that use MPLS to tunnel peering 
traffic from their edge back to a centralized “core” router that speaks BGP.  
Not sure how common this is, I have a very small sample set.  But in this 
example, no matter how diverse the carrier is at L1/L2, your L3 investigation 
will always hit the same much-smaller set of routers.



To directly answer your question, the cost/benefit is driven by the fact that 
running BGP is (relatively) complicated, error-prone and expensive, compared to 
not running BGP.  And those routers running BGP are (broadly speaking) the 
routers controlling inter-carrier traffic.  So the “chokepoints” naturally 
occur as an emergent property of each carrier controlling their own operational 
and financial risk.  Very much depends on the carrier and their operational 
philosophy.  I know nearly nothing about Telia, but Zayo doesn’t (didn’t? did 
they get bought?) have a lot of peering routers – they were historically more 
of an L2 and/or Private-network operator, as I understand it.  (I was never a 
customer, so that’s hearsay.)

-Adam



Adam Thompson
Consultant, Infrastructure Services
[MERLIN]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca<mailto:athomp...@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>



From: NANOG  On Behalf Of PAUL 
R BARFORD
Sent: Tuesday, January 18, 2022 8:49 AM
To: davidbass...@gmail.com
Cc: nanog@nanog.org
Subject: Re: Long hops on international paths



Hello David,



Understanding the physical topology of the network is not​ our objective.  What 
we're trying to understand is the logical topology revealed by traceroute (we 
are well-aware of traceroute limitations) and why a relatively small set of 
routers in different countries tend to have the majority of the international 
connections.  Our expectation was that layer 3 connectivity revealed in 
traceroute to be relatively evenly spread out along coasts and near submarine 
landing points.  We're not seeing that.  So, the question is what is the 
cost/benefit to providers to configure/maintain routes (that include long MPLS 
tunnels) that tend to concentrate international connectivity at a relatively 
small number of routers?



Regards, PB



From: davidbass...@gmail.com<mailto:davidbass...@gmail.com> 
mailto:davidbass...@gmail.com>>
Sent: Tuesday, January 18, 2022 8:22 AM
To: PAUL R BARFORD mailto:p...@cs.wisc.edu>>
Cc: morrowc.li...@gmail.com<mailto:morrowc.li...@gmail.com> 
mailto:morrowc.li...@gmail.com>>; 
nanog@nanog.org<mailto:nanog@nanog.org> 
mailto:nanog@nanog.org>>
Subject: Re: Long hops on international paths



I think a large part of your problem is that you’re using trace route to try 
and determine the full topology of a large complex network.  It won’t show the 
full topology.



On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:

What we're considering specifically are consecutive (layer 3) hops as 
identified by traceroute.  Thus, TTL is decremented by 1 and no more than 1 
(i.e., we have to get full information (not *) from consecutive hops to 
consider the link).  I have asked my colleague to put together a set of 
examples.  We assume that there are multiple layer 1 and 2 link

Re: Long hops on international paths

[PAUL R BARFORD]

Thanks Saku and Michael.

From: Saku Ytti 
Sent: Tuesday, January 18, 2022 9:17 AM
To: Michael Hare 
Cc: PAUL R BARFORD ; Esteban Carisimo 
; nanog@nanog.org ; Fabian 
E. Bustamante 
Subject: Re: Long hops on international paths

On Tue, 18 Jan 2022 at 17:14, Michael Hare  wrote:

> AS3128 runs MPLS and it's probable someone might correct me here, but for a 
> IGP backbone area I think it's common for there to be a full mesh of LSPs via 
> either LDP, RSVP, SR etc.  AS3128 is a small regional and we operate in that 
> way across 60+ nodes.  I don't know if it's common for someone with a global 
> footprint like 1299 to have a contiguous global MPLS backbone, but the point 
> of my reply was to say it's not impossible to think 1299 has a global MPLS 
> mesh between major POPs.

Yes, this is the common case, not an exception.

--
  ++ytti

Re: Long hops on international paths

[PAUL R BARFORD]

Hello David,

Understanding the physical topology of the network is not​ our objective.  What 
we're trying to understand is the logical topology revealed by traceroute (we 
are well-aware of traceroute limitations) and why a relatively small set of 
routers in different countries tend to have the majority of the international 
connections.  Our expectation was that layer 3 connectivity revealed in 
traceroute to be relatively evenly spread out along coasts and near submarine 
landing points.  We're not seeing that.  So, the question is what is the 
cost/benefit to providers to configure/maintain routes (that include long MPLS 
tunnels) that tend to concentrate international connectivity at a relatively 
small number of routers?

Regards, PB

From: davidbass...@gmail.com 
Sent: Tuesday, January 18, 2022 8:22 AM
To: PAUL R BARFORD 
Cc: morrowc.li...@gmail.com ; nanog@nanog.org 

Subject: Re: Long hops on international paths

I think a large part of your problem is that you’re using trace route to try 
and determine the full topology of a large complex network.  It won’t show the 
full topology.

On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
What we're considering specifically are consecutive (layer 3) hops as 
identified by traceroute.  Thus, TTL is decremented by 1 and no more than 1 
(i.e., we have to get full information (not *) from consecutive hops to 
consider the link).  I have asked my colleague to put together a set of 
examples.  We assume that there are multiple layer 1 and 2 links, and possibly 
layer 3 hops masked from traceroute by MPLS.  But what we're seeing in terms of 
hops exposed by traceroute make it look like a single (TTL decremented by 1) 
hop.

I'll post the examples when I get them.

PB

From: morrowc.li...@gmail.com<mailto:morrowc.li...@gmail.com> 
mailto:morrowc.li...@gmail.com>>
Sent: Monday, January 17, 2022 5:13 PM
To: PAUL R BARFORD mailto:p...@cs.wisc.edu>>
Cc: Pengxiong Zhu mailto:pzhu...@ucr.edu>>; 
nanog@nanog.org<mailto:nanog@nanog.org> 
mailto:nanog@nanog.org>>

Subject: Re: Long hops on international paths



On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Dear Pengxiong,

Thanks for your questions:


  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the 
MIDAR alias resolution method to infer IP addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the 
router in question are assigned similar domain names e.g., 
“chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” 
(62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation 
information, and indicates that this router is located in Chicago.  We 
cross-reference with Maxmind where possible.  In this particular case, there is 
the telltale in the use of "chi" in the domain name.
  3.

I think nick's point about ttl expiry and missing some context on topology 
still stands.
I'd be that the paths between 2 continents do not actually land in chicago... 
that you're seeing (or not seeing) missing hops between the coast(s) and 
chicago inside 1299's network in the US.


  1.

Hope that helps.

Regards, PB

From: Pengxiong Zhu mailto:pzhu...@ucr.edu>>
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD mailto:p...@cs.wisc.edu>>
Cc: nanog@nanog.org<mailto:nanog@nanog.org> 
mailto:nanog@nanog.org>>
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP 
address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in 
Chicago? IP-geolocation based on rDNS is not always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is rela

Re: Long hops on international paths

[PAUL R BARFORD]

Hello Saku,

Thank you for the summary.  We're clear about the fact that what we're seeing 
are MLPS paths - that was not in question.  What we are not clear about and the 
reason for the post is why the provider - zayo.telia in this case - would 
decide to configure MPLS paths between Chicago and distant international 
locations.  We assumed we would see hops in traceroute between Chicago and 
coastal locations and then hops that transited submarine infrastructure 
followed by hops to large population centers.

Regards, PB

From: Saku Ytti 
Sent: Tuesday, January 18, 2022 12:50 AM
To: PAUL R BARFORD 
Cc: Lukas Tribus ; Esteban Carisimo 
; nanog@nanog.org ; Fabian 
E. Bustamante 
Subject: Re: Long hops on international paths

1) all (meaning all hitting the zayo.telia) your traceroutes originate
from University in Chicago
2) the zayo.telia device is physically close to the university
3) we should expect physically close-by backbone device to be present
in disproportionate amount of traceroutes
4) almost certainly zayo.telia is imposing the MPLS label of TTL 255,
_NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not
expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you
are not seeing any P routers.

This is not esoteric knowledge, but a fairly basic Internet concept. I
am worried you are missing too much context to produce actionable
output from your work. It might be interesting to see your curriculum,
why this confusion arose, why it seems logical that the reason must be
that almost all waves are terminated there, because it would not seem
logical for people practising in the field who have even cursory
understanding, this implies problems in the curriculum.

On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD  wrote:
>
> Please find the examples for the case of Telia below.
>
> FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
>
>
>
> traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No 
> AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
>
> 1  216.66.30.101  0.365 ms
>
> 2  62.115.49.173  3.182 ms
>
> 3  *
>
> 4  62.115.137.59  17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC 
> -> Chicago, IL, US)
>
> 5  62.115.117.48  59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> 
> Seattle, WA, US)
>
> 6  62.115.171.221  69.993 ms
>
> 7  223.120.6.53  69.378 ms
>
> 8  223.120.12.34  226.225 ms
>
> 9  221.183.55.110  237.475 ms
>
> 10  221.183.25.201  238.697 ms
>
> 11  221.176.16.213  242.296 ms
>
> 12  221.183.36.62  352.695 ms
>
> 13  221.183.39.2  300.166 ms
>
> 14  117.191.8.118  316.270 ms
>
> 15  *
>
> 16  *
>
> 17  *
>
> 18  *
>
> 19  *
>
>
>
>
>
> FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at 
> Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., 
> MAXMIXD: La Crau, FR)
>
> 1  140.192.218.129  0.795 ms
>
> 2  140.192.9.124  0.603 ms
>
> 3  64.124.44.158  1.099 ms
>
> 4  64.125.31.172  3.047 ms
>
> 5  *
>
> 6  64.125.15.65  1.895 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.118.59  99.242 ms[x] (prs-b3-link.ip.twelve99.net., 
> CAIDA-GEOLOC -> Paris, FR)
>
> 8  62.115.154.23  105.214 ms
>
> 9  77.136.10.6  119.021 ms
>
> 10  77.136.10.6  118.830 ms
>
> 11  80.118.89.202  118.690 ms
>
> 12  80.118.89.234  118.986 ms
>
> 13  109.24.108.66  119.159 ms
>
> 14  109.25.215.237  126.085 ms
>
>
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at 
> Depaul University-AS20120) to 84.249.89.93 
> (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI)
>
> 1  140.192.218.129  0.243 ms
>
> 2  140.192.9.124  0.326 ms
>
> 3  64.124.44.158  0.600 ms
>
> 4  *
>
> 5  *
>
> 6  64.125.15.65  1.792 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.123.27  121.199 ms   [x] (hls-b4-link.ip.twelve99.net., 
> CAIDA-GEOLOC -> Helsinki, FI)
>
> 8  *
>
> 9  141.208.193.190  127.723 ms
>
> 10  84.249.89.93  139.051 ms
>
>
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 
> 193.28.231.50 (MAXMIXD: None, HU)
>
> 1  140.192.218.129  0.240 ms
>
> 2  140.192.9.124  0.333 ms
>
> 3  64.124.44.158  0.648 ms
>
> 4  *
>
> 5  64.125.25.75  0.752 ms
>
> 6  64.125.15.65  1.877 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.119.39  123.952 ms   [x] (

Re: Long hops on international paths

[PAUL R BARFORD]

Please find the examples for the case of Telia below.


FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)



traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS 
info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)

1  216.66.30.101  0.365 ms

2  62.115.49.173  3.182 ms

3  *

4  62.115.137.59  17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> 
Chicago, IL, US)

5  62.115.117.48  59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> 
Seattle, WA, US)

6  62.115.171.221  69.993 ms

7  223.120.6.53  69.378 ms

8  223.120.12.34  226.225 ms

9  221.183.55.110  237.475 ms

10  221.183.25.201  238.697 ms

11  221.176.16.213  242.296 ms

12  221.183.36.62  352.695 ms

13  221.183.39.2  300.166 ms

14  117.191.8.118  316.270 ms

15  *

16  *

17  *

18  *

19  *





FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)



traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La 
Crau, FR)

1  140.192.218.129  0.795 ms

2  140.192.9.124  0.603 ms

3  64.124.44.158  1.099 ms

4  64.125.31.172  3.047 ms

5  *

6  64.125.15.65  1.895 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.118.59  99.242 ms[x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC 
-> Paris, FR)

8  62.115.154.23  105.214 ms

9  77.136.10.6  119.021 ms

10  77.136.10.6  118.830 ms

11  80.118.89.202  118.690 ms

12  80.118.89.234  118.986 ms

13  109.24.108.66  119.159 ms

14  109.25.215.237  126.085 ms





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., 
MAXMIXD: Turku, FI)

1  140.192.218.129  0.243 ms

2  140.192.9.124  0.326 ms

3  64.124.44.158  0.600 ms

4  *

5  *

6  64.125.15.65  1.792 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.123.27  121.199 ms   [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC 
-> Helsinki, FI)

8  *

9  141.208.193.190  127.723 ms

10  84.249.89.93  139.051 ms





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 
193.28.231.50 (MAXMIXD: None, HU)

1  140.192.218.129  0.240 ms

2  140.192.9.124  0.333 ms

3  64.124.44.158  0.648 ms

4  *

5  64.125.25.75  0.752 ms

6  64.125.15.65  1.877 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.119.39  123.952 ms   [x] (bpt-b2-link.ip.twelve99.net., **I suspect 
it is in Budapest, HU**)

8  62.115.39.122  117.171 ms

9  88.151.96.148  117.202 ms

10  88.151.96.213  124.787 ms

11  *

12  *

13  *

14  *

15  *





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)

1  140.192.218.129  0.224 ms

2  140.192.9.124  0.545 ms

3  64.124.44.158  0.640 ms

4  *

5  *

6  64.125.15.65  1.786 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.118.247  54.597 ms   [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC 
-> Los Angeles, CA, US)

8  62.115.11.129  55.979 ms

9  *

10  *

11  *

12  *

13  *





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)

1  140.192.218.129  2.277 ms

2  140.192.9.124  0.449 ms

3  64.124.44.158  0.576 ms

4  *

5  *

6  64.125.15.65  1.814 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.114.41  210.056 ms   [x] (snge-b5-link.ip.twelve99.net.,)

8  62.115.177.11  200.840 ms

 9  103.198.140.16  233.636 ms

10  103.198.140.16  232.871 ms

11  103.198.140.171  232.648 ms

12  *

13  *

14  *

15  *

16  *


From: Lukas Tribus 
Sent: Monday, January 17, 2022 1:52 PM
To: PAUL R BARFORD 
Cc: Nick Hilliard ; nanog@nanog.org ; Esteban 
Carisimo ; Fabian E. Bustamante 

Subject: Re: Long hops on international paths

On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD  wrote:
> What we're curious about is why we're seeing a concentration of hops at a 
> small number of routers that appear on international paths.

I suggest you share a few actual examples (IP addresses, traceroutes).

I don't think discussing your conclusion based on data we don't have
makes sense.


Lukas

Re: Long hops on international paths

[PAUL R BARFORD]

What we're considering specifically are consecutive (layer 3) hops as 
identified by traceroute.  Thus, TTL is decremented by 1 and no more than 1 
(i.e., we have to get full information (not *) from consecutive hops to 
consider the link).  I have asked my colleague to put together a set of 
examples.  We assume that there are multiple layer 1 and 2 links, and possibly 
layer 3 hops masked from traceroute by MPLS.  But what we're seeing in terms of 
hops exposed by traceroute make it look like a single (TTL decremented by 1) 
hop.

I'll post the examples when I get them.

PB

From: morrowc.li...@gmail.com 
Sent: Monday, January 17, 2022 5:13 PM
To: PAUL R BARFORD 
Cc: Pengxiong Zhu ; nanog@nanog.org 
Subject: Re: Long hops on international paths



On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Dear Pengxiong,

Thanks for your questions:


  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the 
MIDAR alias resolution method to infer IP addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the 
router in question are assigned similar domain names e.g., 
“chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” 
(62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation 
information, and indicates that this router is located in Chicago.  We 
cross-reference with Maxmind where possible.  In this particular case, there is 
the telltale in the use of "chi" in the domain name.
  3.

I think nick's point about ttl expiry and missing some context on topology 
still stands.
I'd be that the paths between 2 continents do not actually land in chicago... 
that you're seeing (or not seeing) missing hops between the coast(s) and 
chicago inside 1299's network in the US.


  1.

Hope that helps.

Regards, PB

From: Pengxiong Zhu mailto:pzhu...@ucr.edu>>
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD mailto:p...@cs.wisc.edu>>
Cc: nanog@nanog.org<mailto:nanog@nanog.org> 
mailto:nanog@nanog.org>>
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP 
address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in 
Chicago? IP-geolocation based on rDNS is not always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is relatively small (i.e., those that offer global connectivity), 
we assumed that the set of routers that used for international connectivity 
within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, 
the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

--

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside

Re: Long hops on international paths

[PAUL R BARFORD]

Dear Pengxiong,

Thanks for your questions:


  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the 
MIDAR alias resolution method to infer IP addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the 
router in question are assigned similar domain names e.g., 
“chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, which 
provides geolocation information, and indicates that this router is located in 
Chicago.  We cross-reference with Maxmind where possible.  In this particular 
case, there is the telltale in the use of "chi" in the domain name.
  3.

Hope that helps.

Regards, PB

From: Pengxiong Zhu 
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD 
Cc: nanog@nanog.org 
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP 
address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in 
Chicago? IP-geolocation based on rDNS is not always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is relatively small (i.e., those that offer global connectivity), 
we assumed that the set of routers that used for international connectivity 
within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, 
the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

--

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside

Re: Long hops on international paths

[PAUL R BARFORD]

Hello Nick,

I've added my collaborators to this reply - Esteban can comment on your 
observation re. Telia.

TTL is​ decremented on the paths we're analyzing.  What we're curious about is 
why we're seeing a concentration of hops at a small number of routers that 
appear on international paths.  I expected that when we looked at paths between 
e.g., US-Asia, US-Europe, US-South America (considering measurements from Ark 
nodes doing traceroutes to/from those locations), we would see the first 
instances of routers located the US along the west coast, east coast and south 
coast respectively. We did not expect to see Chicago as a first hop location in 
the US and are wondering e.g., if large providers do this to simplify their 
operations.  Hopefully that makes sense.  Any further thoughts are appreciated.

Regards, PB

From: Nick Hilliard 
Sent: Monday, January 17, 2022 12:36 PM
To: PAUL R BARFORD 
Cc: nanog@nanog.org 
Subject: Re: Long hops on international paths

PAUL R BARFORD wrote on 17/01/2022 18:02:
> For example, there is a router operated by Telia (AS1299) in Chicago
> that has a high concentration of such links.

this doesn't appear to match 1299's public network topology:

https://www.teliacarrier.com/our-network.html

Is ttl decrement disabled on the test paths you're measuring?

Broadly speaking, if you have a point-to-point link from one location to
another (or parallel set of links with a common failure path, e.g. waves
on a specific fibre path), there's a single router at each end.

Nick

Long hops on international paths

[PAUL R BARFORD]

Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is relatively small (i.e., those that offer global connectivity), 
we assumed that the set of routers that used for international connectivity 
within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, 
the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

Re: BGP Route Monitoring

[Paul Rolland]

Hello,

On Fri, 7 Jan 2022 09:50:25 +0200
Saku Ytti  wrote:

> On Thu, 6 Jan 2022 at 15:48, Sandoiu Mihai  wrote:
> 
> > I am trying to find a solution that does not require much scripting or
> > customization.  
> 
> Suggestion to run BMP is a fine suggestion. Another option is plain
> old BGP, setup iBGP+best-external (w/ add-path if you may receive >1
> copy from local eBGP neighbours) from these boxes to a collector bgp,
> maybe with a prefix-list filter to send only this prefix. Then have
> the collector box raise an alert when it doesn't receive the route
> from one of them.

What about setting up a machine with exabgp installed, a iBGP session with
the exabgp instance, and a small script parsing the updates received by
exabgp to raise an alarm whenever $CONDITION is met ?

https://github.com/Exa-Networks/exabgp

Best,
Paul


pgpz62pFLxykv.pgp
Description: OpenPGP digital signature

Identifying submarine links via traceroute

[PAUL R BARFORD]

Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying submarine cable infrastructure.

Our interest is in identifying submarine links in traceroute measurements.  
Specifically, for a given end-to-end traceroute measurement, we would like to 
be able to identify when two hops are separated by a submarine cable.  Our 
initial focus has been on inter-hop latency, which can expose long links.  The 
challenge is that terrestrial long-haul links may have the same or longer link 
latencies as short submarine links. So, we're interested in whether there may 
be other features (e.g., persistent congestion, naming conventions in router 
interfaces, peering details, etc.) or techniques that would indicate submarine 
links.

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

Re: Squat space is now being advertised by AS 749 (DoD Network Information Center)

[Paul Ferguson]

Both articles are base don Doug Madory's research:

https://www.kentik.com/blog/wait-did-as8003-just-disappear/

Cheers,

- ferg


On 9/10/21 5:26 PM, Daniel Lacey wrote:


Just saw an article in the Washington Post explaining what went on…

It was a follow up to the Apr 24 and 26 articles…

I don’t have a link without a subscription….

Basically, unused IPv4 addresses from DOD were being transferred to 
Global Resource  Systems. It was transferred back today.This is some 
pilot program for network resilience by the Pentagon unit Defense 
Digital Service.


I don’t know if this is a smoke screen or exactly what “they” say it is…
Just trying to fill in the blanks…

On Sep 10, 2021, at 15:40, Compton, Rich A  
wrote:




Hi, this week it looks like the DoD owned squat space that was 
previously advertised by AS 8008 (a shadow company called Global 
Resource Systems, 
seehttps://apnews.com/article/technology-business-government-and-politics-b26ab809d1e9fdb53314f56299399949 
<https://apnews.com/article/technology-business-government-and-politics-b26ab809d1e9fdb53314f56299399949>) 
is now being advertised by AS 749 (DoD Network Information Center).  
Does anyone have any idea why this change was made?  Is the DoD 
planning on actually legitimately putting services on the space soon 
instead of using it as a giant honeypot?  Or maybe even selling it?


Thanks,

Rich




--
Paul Ferguson
Tacoma, WA  USA
Illegitimi non carborundum.

Re: netflow in the core used for surveillance

[Paul Ebersman]

randy> 
https://www.vice.com/en/article/jg84yy/data-brokers-netflow-data-team-cymru

randy> at, comcast, ... zayo, please tell us you do not do this.


aaron> You know they do.

No, you don't know that.

The above all certainly collect this info. Not all sell it to anyone who
asks.

Re: SITR/SHAKEN implementation in effect today (June 30 2021)

[Paul Timmins]

Fun part is that just because it's a telnyx number with a checkmark, it 
doesn't mean the call came from Telnyx, just that the call came from a 
carrier that gave the call attestation A. As the carrier, we can see who 
signed the call (it's an x509 certificate, signed by the STI-PA, with 
the carrier's name and OCN in it) and hold them accountable for the 
traffic, which is huge.


But that's where the confusion will lie - a customer might say well this 
is a verizon wireless number, i'll yell at them! But the actual call 
came in through Lumen, and they're the ones who can stop it. A carrier 
can see the cert, but you can just get the verstat flag from the 
P-Asserted-Identity field in the call to the handset and see that it 
passed the tests for attestation A.


Just because you don't see a checkmark doesn't mean signatures aren't 
happening. Attestation B and C aren't displayed on the handset (but are 
seen in the carrier's systems) and most androids don't have a way to 
display stir/shaken stuff yet. T-Mobile doesn't send the verstat header 
to handsets they don't verify as s/s compliant (usually only ones they 
sell). My trick was to sim swap into an iphone for a day, then back to 
my android which started displaying the verification after that.


It's all new, but just because you don't see it doesn't mean it's not 
there. Report the calls to your carrier, they have new tools to track 
down the misbehavior.


On 7/2/21 8:32 AM, Nick Olsen wrote:
Not all have implemented it yet. But if you haven't. You were supposed 
to implement some kind of robo calling mitigation plan (Or atleast 
certify that you have one). At $dayjob we're fully deployed (inbound 
and outbound).


I received my first ever STIR/SHAKEN signed (iPhone Check mark, highly 
scientific) spam call on my personal Cell phone on 6/30. It was a 
Telnyx number. Had the call terminated to $dayjob network. I fully 
would have collected all various information and ticketed it with Telnyx.


Time will tell how truly effective this is. But we have better 
originating information now (breadcrumbs) to follow back to the source.


On Thu, Jul 1, 2021 at 5:42 PM Andreas Ott > wrote:




On Thu, Jul 1, 2021 at 12:56 PM Keith Medcalf mailto:kmedc...@dessus.com>> wrote:

... and the end carrier is making money for terminating them. 



Survey (of n=1) says: nothing has changed, aka the new technology
is not working. I just received the same kind of recorded message
call of "something something renew auto warranty" on my AT
u-Verse line. This time when I called back the displayed caller ID
number it was ring-no-answer, versus the previous "you have
reached a number that is no longer in service". By terminating the
call the carrier made probably more money than it would cost them
to enforce the new rules.

Other than the donotcall.gov  portal, is
there a new way to report the obvious failure of STIR/SHAKEN?

-andreas

Re: SITR/SHAKEN implementation in effect today (June 30 2021)

[Paul Timmins]

On 7/1/21 3:53 PM, Keith Medcalf wrote:

And this is why this problem will not be solved.  The "open relay" is making money from processing 
the calls, and the end carrier is making money for terminating them.  Until fine(s) -- hopefully millions of 
them, one for each improperly terminated call, together with jail time for the CEO of the company for 
"conspiracy to commit fraud" --  and EACH of the fines is EQUAL OR GREATER than the total yearly 
worldwide REVENUE of that end carrier, they will not have any impetus to "fix" the problem.


How about 47 CFR 64.1200(k)(4)?

(4) A provider may block voice calls or cease to accept traffic from an 
originating or intermediate provider 
 
without liability under the Communications Act or the Commission's rules 
where the originating or intermediate provider 
, 
when notified by the Commission, fails to effectively mitigate illegal 
traffic within 48 hours or fails to implement effective measures to 
prevent new and renewing customers 
 
from using its network to originate illegal calls. Prior to initiating 
blocking, the provider shall provide the Commission with notice and a 
brief summary of the basis for its determination that the originating or 
intermediate provider 
 
meets one or more of these two conditions for blocking.


ie: "You're not really a phone company anymore, says the rest of the PSTN"

https://www.law.cornell.edu/cfr/text/47/64.1200

Re: SITR/SHAKEN implementation in effect today (June 30 2021)

[Paul Timmins]

On 6/30/21 2:56 PM, Michael Thomas wrote:
Just because you can know (fsvo "know") that a call is allowed to 
assert a number doesn't change anything unless other actions are 
taken. With DKIM which is far simpler than STIR it would require 
reputation systems that don't seem to have been deployed, submission 
auth which thankfully was deployed, policy enforcement (ie ADSP) which 
is not deployed, and user indicators which are sporadically deployed.


In any indication, the carrier closest to the originator is signing the 
call metadata with their digital certificate. While this won't mean much 
to the active user, for those tracking down robocalls, this is the holy 
grail - finding the carrier who is letting the calls into the network 
and being able to reach out to them to stop the abusive/illegal traffic.


That it might say we've taken the time to verify the end user is who 
they say they are is just icing on the cake. The goal is to make the 
calls accountable to someone, which despite the patchwork of systems in 
the US that might prevent the signature from coming through, can help a 
lot since the biggest wholesalers have implemented it (Inteliquent and 
Lumen among many others)


The other big deal is that now all carriers are actually expected to 
police their network for spoofed callers who are exhibiting robocalling 
behavior. This is a big deal! For the first time, carriers are going to 
be held responsible for proactively finding the abuse, and showing what 
their plans are to do such a thing, and sharing information with each 
other (via the FCC) who can be contacted to chase down robocall traffic 
if another carrier sees it.


Given the giant security holes caused by solving the wrong problem (ie 
trying to authenticate the e.164 address rather than the originating 
domain) it's just going to push spammers to exploit those holes. It's 
very much to be seen whether victory can be declared, IMO.


Fortunately, positive identification of the caller isn't the intent. 
Preventing people from pretending to be the IRS is the intent.


-Paul

Re: an IP hijacking attempt

[Paul Emmons]

RPKI can be very useful to mitigate an attempt.

I used to process IP LOAs all the time.  I never saw a RR attached but 
usually we did a check against the RIR just to make sure (because we 
made access-list per interface as well)


On 3/9/2021 1:42 PM, Mel Beckman wrote:
Not everyone uses RRs, and there is also the possibility that their 
upstream would register it. Having an RR doesn’t seem definitive 
either way. I can see reasons to wait on the RR until  ready to 
receive traffic.


-mel via cell


On Mar 9, 2021, at 11:14 AM, Brian Turnbow  wrote:


If they had a route record that was close, I Would give them the 
benefit of doubt.
They do not however as the only records start with 217. And our IPs 
are 45.


So It Is very strange. Would you send a LOA without a route record?

Brian Turnbow

*Da:* Mel Beckman 
*Inviato:* martedì 9 marzo 2021 19:17
*A:* Brian Turnbow
*Cc:* North American Network Operators' Group
*Oggetto:* Re: an IP hijacking attempt

It could just be a typo on the LOA. It seems unlikely any ISP would 
approve a forged LOA that could readily be debunked by contacting the 
IP space owner. The whole point of LOA’s is to facilitate this 
verification.


-mel via cell

> On Mar 9, 2021, at 10:01 AM, Brian Turnbow via NANOG 
 wrote:

>
> Hello everyone,
>
> We received a strange request that I wanted to share.
> An email was sent to us asking to confirm a LOA from a diligent ISP.
> The Loa was a request to open bgp for an AS , that is not ours, to 
announce a /23 prefix that is ours.
> So basically this entity sent to their upstream a request to 
announce a prefix from one our allocated ranges.
> We have the allocation correctly registered and ROAs in place , but 
it is worrisome that someone would attempt this.
> Obviously we have informed the ISP that the LOA is not valid and 
are trying to contact the originating party.
> Aside from RIRs for the offending AS and our IPs,  Is there 
anywhere to report this type of activity?
> We have dealt with hijacking technically speaking in the past but 
this is the first time, to my knowledge, of someone forging a LOA 
with our IPs.

>
> Thanks in advance for any advice
>
> Brian
>
> P.S. a big thanks to Chris for checking the boxes before activating 
the filter if you are on the list!

>
>
>
>

Re: Famous operational issues

[Paul Ebersman]

warren> 2: A somewhat similar thing would happen with the Ascend TNT
warren> Max, which had side-to-side airflow. These were dial termination
warren> boxes, and so people would install racks and racks of them. The
warren> first one would draw in cool air on the left, heat it up and
warren> ship it out the right. The next one over would draw in warm air
warren> on the left, heat it up further, and ship it out the
warren> right... Somewhere there is a fairly famous photo of a rack of
warren> TNT Maxes, with the final one literally on fire, and still
warren> passing packets.

The Ascend MAX (TNT was the T3 version, max took 2 T1s) was originally
an ISDN device. We got the first v.34 rockwell modem version for
testing. An individual card had 4 daughter boards. They were burned in
for 24 hours at Ascend, then shipped to us. We were doing stress testing
in Fairfax VA. Turns out that the boards started to overheat at about 30
hours and caught fire a few hours after that... Completely melted the
daughterboards. They did fix that issue and upped the burnin test period
to 48 hours.

And yeah, they vented side to side. They were designed for enclosed
racks where are flow was forced up. We were colocating at telco POPs so
we had to use center mount open relay racks. The air flow was as you
describe. Good time. Had by all...

Both we (UUNET, for MSN and Earthlink) and AOL were using these for
dialup access. 80k ports before we switched to the TNTs, 3+ million
ports on TNTs by the time I stopped paying attention.

Re: Famous operational issues

[Paul Ebersman]

jlewis> This reminds me of one of the Sprint CO's we were colo'd in.

Ah, Sprint. Nothing like using your railroad to run phone lines...
Our routers in San Jose colo were black from the soot of the trains.

Fondly remember a major Sprint outage in the early 90s. All our data
circuits in the southeast went down at once and there were major voice
outages in the entire southeast.

Turns out a storm caused a mudslide which in turn derailed a train
carrying toxic waste, resulting in a wave of 6-10' of toxic mud taking
out the Spring voice pop for the whole southeast, because it was
conveniently located right on said railroad tracks.

We were a big enough customer that PLSC in Atlanta gave us the real
story when we asked for an ETA on repair. They couldn't give us one
immediately until the HAZMAT crew let them in. Turned out to be a total
loss of all gear.

They yanked every tech east of the Misssissippi and a 7ESS was Fedex
overnighted (stolen from some customer in the middle east?) and they had
to rebuild everything.

Was down less than 10 days. Good times.

Re: Alexandria Ocasio-Cortez' Office is on NANOG?? Or, what is the policy about sharing email offlist?

[Paul Timmins]

The list has public archives. Draw your own conclusions on the policy.

https://mailman.nanog.org/pipermail/nanog/

On 1/18/21 2:40 PM, Anne P. Mitchell, Esq. wrote:
Not under that impression at all. That's very different from "what is 
the policy" - at least in the groups I run, if the policy is "no 
sharing offlist" and then someone does, there are consequences for 
that someone.

Anne

--
Anne P. Mitchell,  Attorney at Law
Dean of Cyberlaw & Cybersecurity, Lincoln Law School
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam law)
Board of Directors, Denver Internet Exchange
Chair Emeritus, Asilomar Microcomputer Workshop
Former Counsel: Mail Abuse Prevention System (MAPS)

Re: Parler

[Paul Timmins]

"You have to let your customer's services contain death threats against 
the owner of your company or we'll blacklist you" is the wildest take of 
2021 yet.


Blocking Amazon because of who they allow to remain a customer is 
something I wholeheartedly encourage my competitors to do.


On 1/12/21 9:29 AM, Kevin McCormick wrote:


Imagine if Tier 1 ISPs had a censorship free clause that required 
companies like Twitter, Facebook, and Amazon to provide services free 
of censorship or have IP blocks blackholed. They would lose hundreds 
of millions of dollars per day. I bet they would reverse their tone in 
a hurry.


https://www.seattletimes.com/seattle-news/idaho-internet-provider-to-block-facebook-twitter-over-their-trump-bans/

Thank you,

Kevin McCormick

*From:*NANOG  *On Behalf 
Of *mark seery

*Sent:* Sunday, January 10, 2021 8:06 PM
*To:* K. Scott Helms 
*Cc:* NANOG Operators' Group 
*Subject:* Re: Parler

I assume multiple networks/ ISPs that have acceptable use policies 
that call out criminality and incitement to violence, for example:


https://www.xfinity.com/support/articles/comcast-acceptable-use-policy

Have these AUPs been invoked previously for these reasons, or would 
that be new territory?


Sent from Mobile Device



On Jan 10, 2021, at 2:52 PM, K. Scott Helms
mailto:kscott.he...@gmail.com>> wrote:



Right, it's not a list for content hosting.

Scott Helms

On Sun, Jan 10, 2021, 5:42 PM mailto:sro...@ronan-online.com>> wrote:

No, this is a list for Network Operators.

Sent from my iPhone



On Jan 10, 2021, at 5:32 PM, K. Scott Helms
mailto:kscott.he...@gmail.com>>
wrote:



This is a list for pushing bits.  The fact that many/most
of us have other businesses doesn't make this an
appropriate forum for SIP issues (to use my own work as an
example).

On Sun, Jan 10, 2021, 4:52 PM mailto:sro...@ronan-online.com>> wrote:

This is a list for Network Operators, AWS certainly
operates networks.

Sent from my iPhone



On Jan 10, 2021, at 4:27 PM, K. Scott Helms
mailto:kscott.he...@gmail.com>> wrote:



No,

It really does not.  Section 230 only applies to
publishers, and not to network providers.  If this
were a cloud hosting provider list then you'd be
correct, but as a network provider's list it does
not belong here.


Scott Helms

On Sun, Jan 10, 2021 at 3:21 PM Lady Benjamin PD
Cannon mailto:b...@6by7.net>> wrote:

As network operations and
compute/cloud/hosting operations continue to
coalesce, I very much disagree with you. 
Section 230 is absolutely relevant, this
discussion is timely and relevant, and it
directly affects me as both a telecom and
cloud compute/services provider.

—L.B.

Lady Benjamin PD Cannon, ASCE

6x7 Networks & 6x7 Telecom, LLC

CEO

b...@6by7.net 

"The only fully end-to-end encrypted global
telecommunications company in the world.”

FCC License KJ6FJJ









On Jan 10, 2021, at 12:13 PM, K. Scott
Helms mailto:kscott.he...@gmail.com>> wrote:

It's not, and frankly it's disappointing
to see people pushing an agenda here.


Scott Helms


On Sun, Jan 10, 2021 at 9:37 AM
mailto:sro...@ronan-online.com>> wrote:


NANOG is a group of Operators,
discussion does not have to be about
networking. I have already explained
how this represents a significant
issue for Network Operators.

On Jan 10, 2021, at 9:09 AM, Mike
Bolitho mailto:mikeboli...@gmail.com>> wrote:


It has nothing to do with networking.
Their decision was necessarily
political. If you can specifically
bring up an issue, beyond speculative,
on how their 

RE: Are the days of the showpiece NOC office display gone forever?

[Paul Amaral via NANOG]

We used to have some CRTs with MRTG running in the late 90’s 

 

P

 

From: NANOG  On Behalf Of Eric Kuhnke
Sent: Wednesday, December 16, 2020 3:50 PM
To: nanog@nanog.org list 
Subject: Are the days of the showpiece NOC office display gone forever?

 

With the covid19 situation, obviously lots of ISPs have their NOC personnel 
working from home, with VPN (or remote desktop) access to all the internal 
tools, VoIP at home, etc.

 

In the traditional sense, by "showpiece NOC" I mean a room designed for the 
purpose of having large situational awareness displays on a wall, network 
weathermaps and charts, alerting systems, composed of four or more big flat 
panel displays. Ideally configured to be actually useful for NOC purposes and 
also something impressive looking for customer tours.

 

To what extent potential customers find that sort of thing to be a signifier of 
seriousness on the part of an ISP, I suppose depends on what sort of customers 
they are, and their relative degree of technical sophistication.

 

Are the days of such an environment gone forever? 

 

 

 

Re: Global Peer Exchange

[Paul Emmons]

> You take down a 10g connection and they bill each side $.2 a meg, 95th
> percintile billing.  VLAN between the two sites. Both sites have to have a
> different AS number.  So if you want to move 1g of data, 95th percentile,
> between 2 datacenters I guess it has some utility at $400 a gig effective
> pricing.   I can't beleive it is a great money maker for them. Oh and it's
> Cogent and they say they can't give you above 1500 mtu.

~P

Re: AT - INET Data Caps

[Paul Emmons]

Yes this is common business practice for almost all of the MSOs.

On Mon, Nov 30, 2020 at 9:45 AM Thomas Yarger 
wrote:

> Hello All,
>
> This past week when I was helping my father perform some home networking,
> I called AT to get a newer Arris router and they mentioned that if I were
> to upgrade his service, he would fall under a 1 TB data cap for home
> internet. Is this just in FL or have others seen similar restrictions with
> AT? Thanks!
>
> --
> Thanks,
>
> Thomas Yarger
>
>

Re: AFRINIC IP Block Thefts -- The Saga Continues

[Paul Nash]

Any idea of the outcome?

> On Nov 17, 2020, at 4:54 PM, Valdis Klētnieks  wrote:
> 
> On Tue, 17 Nov 2020 10:02:01 -0800, Jay Hennigan said:
> 
>> In the old days on the NANAE newsgroup, such bogus threats of legal
>> action were categorized as one calling their "cartooney". People who
>> huff and puff and threaten to sue rarely do so. If someone actually
>> plans on suing you, your first hint is typically a knock on the door by
>> a process server, not repeated threats in an online forum.
> 
> Right.  The thing is that unless you're party to the lawsuit, you don't
> know if a process server has been involved.
> 
> Somebody else replied by private email and pointed where the AfriNIC
> CEO wrote that they had, in fact, actually been sued.   So whatever one
> might think of Elad Cohen, he's apparently not a cartooney.

Fwd: Phoenix-IX Contact

[Paul Emmons]

still trying to post . . .


 Forwarded Message 
Subject:Re: Phoenix-IX Contact
Date:   Mon, 16 Nov 2020 13:15:34 -0700
From:   Paul Emmons 
To: nanog@nanog.org



Hello All!

I've been out of the loop here and but have some updates.

There was a change last spring and I moved on to other projects.  But 
that hasn't worked out for the IX.


I  have regained access to all of the elements, including the email and 
voip.  Let me reach out to each of you offline in the coming days.


I have been able to reach out to a few locals that are willing to help 
get the project back up to where it needs to be.


If I haven;t reached out to you in the next 48 hours or you have 
something urgent, please reach out to me here (my personal email) or via 
the Phoenix-IX Contacts


peer...@phoenix-ix.net

+1 602 688-6414

~Paul Emmons
On 11/16/2020 12:23 PM, Neil Hanlon wrote:
While I agree it is objectively irresponsible to abandon a project 
without passing it to another, I think that possibly in this situation 
we don't know all the details?


2020 has been a difficult year for everyone. Perhaps Paul (and 
whomever else may be responsible for Phoenix-IX) were subject to 
things this year beyond their control which led them to be unable to 
work on the project and unable to transfer it, either.. unfortunate, 
yes.. but not malicious surely.


If Paul _is_ reading these messages.. I think support is the best path 
forward.. If there are things that can be done to assist/take over the 
IX... maybe that would help (as you, Kate, had offered towards the 
beginning of this all). Though of course, the first step is _reaching_ 
them... Maybe this can be turned into a "win" for everyone. So: 
Paul/Phoenix-IX -- let the NANOG community know how they/we can help.


--
Neil

On Mon, Nov 16, 2020 at 2:05 PM Kate Gerry <mailto:kge...@outlook.com>> wrote:


An update on my side, we reached out to PhoenixNAP, one of the
Phoenix-IX's vendors.

PhoenixNAP reached out all of their contacts at Phoenix-IX and
have received no response. They are in as much of the dark as the
rest of us. I feel like I'm on the Ghost ship Phoenix-IX.

What I don't understand, is how somebody could abandon a project
without passing it to another person or entity. This is extremely
irresponsible.

—
Kate


On Nov 12, 2020, at 05:11, Marcus Josephson mailto:mjoseph...@inap.com>> wrote:

I tried to get a link to PHX-IX for months. Never heard back from
them, went with Digital Realty Phoenix
*From:*NANOG mailto:nanog-bounces+mjosephson=inap@nanog.org>>*On Behalf
Of*Kate Gerry
*Sent:*Tuesday, November 10, 2020 11:06 AM
*To:*Matt Hoppes mailto:mattli...@rivervalleyinternet.net>>
*Cc:*nanog@nanog.org <mailto:nanog@nanog.org> list
mailto:nanog@nanog.org>>
*Subject:*Re: Phoenix-IX Contact
Matt,
I am running on a huge assumption here, but I think Phoenix-IX
runs on donated infrastructure. From what I recall, there was
only an NRC to join Phoenix-IX.
And in regards to Walt's suggestion, it looks like HE already
started one with Stellar Technologies. https://48ix.net
<https://48ix.net/> but it is only in a single facility. So until
that IX grows, both in peers and footprint, I'm stuck on Phoenix-IX.
I have wondered what happens if a participant storms the IX. Will
somebody appear? Because attempts to reach their NOC/peering
handles has resulted in a lack of response.
I also wonder how the other Ninja-IX exchanges are running, I
haven't heard anything about them, is there the same lack of
communication? Or do those have a local staff?
—
Kate


On Nov 10, 2020, at 06:15, Matt Hoppes
mailto:mattli...@rivervalleyinternet.net>> wrote:
How is the IX still running?  Surely someone must be paying
colo rent?

On 11/10/20 9:03 AM, Eric Kuhnke wrote:

Always a good time for network operators to consider the
risks of having any one person as a single point of
failure for something kind of important:
https://en.wikipedia.org/wiki/Bus_factor
<https://en.wikipedia.org/wiki/Bus_factor>
Disaster recovery and continuity of business plans should
always include the concept of what if some percentage of
the key team members were to be suddenly unavailable
permanently (the Malaysian airline incident, for example).
On Mon, Nov 9, 2020 at 8:08 PM Kate Gerry
mailto:kge...@outlook.com
<mailto:kge...@outlook.com%20%3cmailto:kge...@outlook.com>>>
wrote:
   Is there anybody else even there? I thought that it
was all Paul's show!
   If I was able to (as in, had access to), I would offer
to help/run
   

Re: Phoenix-IX Contact

[Paul Emmons]

Hello All!

I've been out of the loop here and but have some updates.

There was a change last spring and I moved on to other projects. But 
that hasn't worked out for the IX.


I  have regained access to all of the elements, including the email and 
voip.  Let me reach out to each of you offline in the coming days.


I have been able to reach out to a few locals that are willing to help 
get the project back up to where it needs to be.


If I haven;t reached out to you in the next 48 hours or you have 
something urgent, please reach out to me here (my personal email) or via 
the Phoenix-IX Contacts


peer...@phoenix-ix.net

+1 602 688-6414

~Paul Emmons
On 11/16/2020 12:23 PM, Neil Hanlon wrote:
While I agree it is objectively irresponsible to abandon a project 
without passing it to another, I think that possibly in this situation 
we don't know all the details?


2020 has been a difficult year for everyone. Perhaps Paul (and 
whomever else may be responsible for Phoenix-IX) were subject to 
things this year beyond their control which led them to be unable to 
work on the project and unable to transfer it, either.. unfortunate, 
yes.. but not malicious surely.


If Paul _is_ reading these messages.. I think support is the best path 
forward.. If there are things that can be done to assist/take over the 
IX... maybe that would help (as you, Kate, had offered towards the 
beginning of this all). Though of course, the first step is _reaching_ 
them... Maybe this can be turned into a "win" for everyone. So: 
Paul/Phoenix-IX -- let the NANOG community know how they/we can help.


--
Neil

On Mon, Nov 16, 2020 at 2:05 PM Kate Gerry <mailto:kge...@outlook.com>> wrote:


An update on my side, we reached out to PhoenixNAP, one of the
Phoenix-IX's vendors.

PhoenixNAP reached out all of their contacts at Phoenix-IX and
have received no response. They are in as much of the dark as the
rest of us. I feel like I'm on the Ghost ship Phoenix-IX.

What I don't understand, is how somebody could abandon a project
without passing it to another person or entity. This is extremely
irresponsible.

—
Kate


On Nov 12, 2020, at 05:11, Marcus Josephson mailto:mjoseph...@inap.com>> wrote:

I tried to get a link to PHX-IX for months. Never heard back from
them, went with Digital Realty Phoenix
*From:*NANOG mailto:nanog-bounces+mjosephson=inap@nanog.org>>*On Behalf
Of*Kate Gerry
*Sent:*Tuesday, November 10, 2020 11:06 AM
*To:*Matt Hoppes mailto:mattli...@rivervalleyinternet.net>>
*Cc:*nanog@nanog.org <mailto:nanog@nanog.org> list
mailto:nanog@nanog.org>>
*Subject:*Re: Phoenix-IX Contact
Matt,
I am running on a huge assumption here, but I think Phoenix-IX
runs on donated infrastructure. From what I recall, there was
only an NRC to join Phoenix-IX.
And in regards to Walt's suggestion, it looks like HE already
started one with Stellar Technologies. https://48ix.net
<https://48ix.net/> but it is only in a single facility. So until
that IX grows, both in peers and footprint, I'm stuck on Phoenix-IX.
I have wondered what happens if a participant storms the IX. Will
somebody appear? Because attempts to reach their NOC/peering
handles has resulted in a lack of response.
I also wonder how the other Ninja-IX exchanges are running, I
haven't heard anything about them, is there the same lack of
communication? Or do those have a local staff?
—
Kate


On Nov 10, 2020, at 06:15, Matt Hoppes
mailto:mattli...@rivervalleyinternet.net>> wrote:
How is the IX still running?  Surely someone must be paying
colo rent?

On 11/10/20 9:03 AM, Eric Kuhnke wrote:

Always a good time for network operators to consider the
risks of having any one person as a single point of
failure for something kind of important:
https://en.wikipedia.org/wiki/Bus_factor
<https://en.wikipedia.org/wiki/Bus_factor>
Disaster recovery and continuity of business plans should
always include the concept of what if some percentage of
the key team members were to be suddenly unavailable
permanently (the Malaysian airline incident, for example).
On Mon, Nov 9, 2020 at 8:08 PM Kate Gerry
mailto:kge...@outlook.com
<mailto:kge...@outlook.com%20%3cmailto:kge...@outlook.com>>>
wrote:
   Is there anybody else even there? I thought that it
was all Paul's show!
   If I was able to (as in, had access to), I would offer
to help/run
   with the IX. I may live in California, but it's a
realistic car trip
   back and forth to Phoenix.
   --
   Kate

  

Re: AFRINIC IP Block Thefts -- The Saga Continues

[Paul Nash]

If you don’t have  coherent argument, take Trump’s approach with an incoherent 
ad-hominem attack.

I have been filling this issue with a lot of interest, and to date you have 
offered no evidence of anything, apart from your ability to spew vitriol.



> On Nov 16, 2020, at 10:04 AM, Elad Cohen  wrote:
> 
> Tom,
> 
> Until today all I wrote was facts and evidence, in the contrary to 
> Pore-spilling Ronald. When Ronald keeps defaming me here non-stop, Yes my 
> full right is to sue him, even if you prefer that my blood will be shed here 
> by his filthy soul and pores. And you can be sure that I will respond to any 
> of his defamation messages.
> 
> "No value to the community" - there is value to the community, anyone which 
> is following Pore-spilling Ronald is following the ancient snake, it is not 
> written in Old Testament, it is written in Old Scripture, go ahead and check, 
> Ronald visual appearance match one by one to the lawless one description 
> (lawless one is a term of New Testament) according to Old Scripture.
> 
> Go swim in Ronald juicy pores if you would like.
> 
> 
> From: Tom Beecher 
> Sent: Monday, November 16, 2020 4:54 PM
> To: Elad Cohen 
> Cc: nanog@nanog.org ; adm...@nanog.org 
> Subject: Re: AFRINIC IP Block Thefts -- The Saga Continues
>  
> I would like to formally request that Mr. Cohen's privileges to post to this 
> list be revoked, or otherwise curtailed.
> 
> It's one thing to dispute facts with evidence, or generally disagree on a 
> topic. However , threats of legal action and personal attacks citing Old 
> Testament mumbo jumbo, while creative, provide no value to the community. 
> 
> As Mr. Herrin stated well, we are all swimming in enough nutter butter 
> conspiracy theory nonsense every day. I hope we don't normalize it here too. 
> 
> On Mon, Nov 16, 2020 at 4:24 AM Elad Cohen  wrote:
> 
> If AfriNIC says your "purchases" are misappropriation you'll have to do a lot 
> better than conspiracy theories and phrenology in counter argument.
> 
> 
> Why are you using the word "purchases" with quotation marks, it seems that 
> you are a victim of your next paragraph, and I'm writing it with all due 
> respect.
> 
> Did I start legal proceedings with AfriNIC with conspiracy theories or with 
> facts and data?
> 
> Phrenology (and racism) is the field of Coconut Guilmette, according to his 
> own quotes right here in Nanog.
> 
> "If AfriNIC says" - AfriNIC, nor Spamhaus, nor Ops-Trust, nor MyBroadband, 
> are not the word of god and are not above law and justice.
> 
> To remind to everyone: AfriNIC filed a police complaint on themselves. And 
> AfriNIC CEO lied to the community in the following link when he wrote that I 
> was given a chance to respond when in reality all my emails were ignored. And 
> AfriNIC CEO is intentionally hiding from the community that any AfriNIC 
> policy is applied also to any legacy netblock that even don't have a signed 
> RSA (according to the legal proceedings against AfriNIC, and in contradiction 
> to AfriNIC "Legacy Resource Holders" webpage: 
> https://afrinic.net/membership/legacy-resource ), it have implications on 
> each and every legacy resource holder all over the world (that a RIR can just 
> delete a legacy netblock).
> 
> https://lists.afrinic.net/pipermail/community-discuss/2020-January/003458.html
>  - "We are also ensuring that the current holder/contact of the resources are 
> provided with the opportunity of proving their ownership."
> 
> 
> Besides the above, AfriNIC also have a financial motive in their actions, 
> approximately up to more $4M per year from assets that they illegaly and 
> unjustifiably took from me (by sub-allocating them to AfriNIC members), while 
> writing lies to their community and playing a long with the "community 
> pressure" game.
> 
> 
> I find the actions of AfriNIC to be very dangerous, because they are not 
> based on facts and data and law and justice, but only on community pressure. 
> Any network operator, that can elevate himself/herself from bad habits (like 
> enjoying seeing blood spilled and jealousy), would know that today it is me 
> but tomorrow it can be you.
> 
> 
> 
> From: NANOG  on behalf of William 
> Herrin 
> Sent: Monday, November 16, 2020 10:00 AM
> Cc: nanog@nanog.org 
> Subject: Re: AFRINIC IP Block Thefts -- The Saga Continues
>  
> On Sun, Nov 15, 2020 at 10:58 PM Elad Cohen  wrote:
> > Anyone that is interested to receive any answer from me, please email me 
> > directly, I will say that Ronald Guilmette is intentionally spreading lies, 
> > and for the sake of Nanog community I will not reply to him over and over 
> > in the same coin, I was gladly interested in the past to share all the 
> > information (including AfriNIC legal proceedings) with a person respected 
> > by the Nanog community (and I'm still interested to do so today), such as 
> > William Herrin, or to anyone else respected by the Nanog community.
> 
> Ugh. I don't suppose I can 

PLEASE CHECK THE REPLY EMAIL ADDRESS -- Re: QB server hiccups

[Paul Nash]

Typo in the first version copied this to a mailing list.

I sent a newer version shortly after copied to Brian instead :-)

Please delete the earlier one & only reply to the later one.

Thanks

paul

> On Oct 22, 2020, at 1:19 PM, Paul Nash  wrote:
> 
> After an outage yesterday, I am trying to streamline and simplify the St 
> Felix QB setup to make it more reliable and easier to administer.
> 
> The most critical factor that influences the overall design is the realistic 
> maximum number of simultaneous users.
> 
> If we can live with a maximum of two users logged on at any one time, I can 
> simplify the system dramatically, which will streamline operations and 
> improve reliability.  
> 
> If we need to have more than two users logged at any time, then we need the 
> current setup, with its various woes.  The problem are not insurmountable, 
> but the system is more complex.  FWIW, a quick check looks like Shan has been 
> the only active user for the past couple of months, but I may be wrong.
> 
> Either way, I need to overhaul some parts of the current setup, which was 
> thrown together in an enormous rush.  Once I know what parts we are going to 
> follow, I will make appropriate plans and let everyone know.
> 
> Yesterday’s problems were caused by the combination of Techsoup and 
> Microsoft.  Licenses that I ordered and paid for were not delivered 
> correctly, leading to the QB server shutting down at an inopportune time.  
> After restoring the server, reverting to trial licenses, I spoke to MS tech 
> support, who suggested that we start all over again purchasing the TS 
> licenses.  They also suggested that we just buy them directly from MS at full 
> retail price, which is not much more that the Techsoup price.
> 
> Right now, we should be good for a couple of months, but I want to prevent 
> any similar issues in future, and even the regular jumping-through-hoops that 
> I have been doing up until now to keen the current system running.
> 
> The first issue to resolving this and improving long-term reliability, is to 
> decide how many simultaneous users we realistically need.  The rest will flow 
> from that.
> 
> Regards
> 
>   paul

APOLOGIES: QB server hiccups

[Paul Nash]

Autocorrect changed a misspelled recipient to “nanog”.

paul (grovelling for forgiveness)

QB server hiccups

[Paul Nash]

After an outage yesterday, I am trying to streamline and simplify the St Felix 
QB setup to make it more reliable and easier to administer.

The most critical factor that influences the overall design is the realistic 
maximum number of simultaneous users.

If we can live with a maximum of two users logged on at any one time, I can 
simplify the system dramatically, which will streamline operations and improve 
reliability.  

If we need to have more than two users logged at any time, then we need the 
current setup, with its various woes.  The problem are not insurmountable, but 
the system is more complex.  FWIW, a quick check looks like Shan has been the 
only active user for the past couple of months, but I may be wrong.

Either way, I need to overhaul some parts of the current setup, which was 
thrown together in an enormous rush.  Once I know what parts we are going to 
follow, I will make appropriate plans and let everyone know.

Yesterday’s problems were caused by the combination of Techsoup and Microsoft.  
Licenses that I ordered and paid for were not delivered correctly, leading to 
the QB server shutting down at an inopportune time.  After restoring the 
server, reverting to trial licenses, I spoke to MS tech support, who suggested 
that we start all over again purchasing the TS licenses.  They also suggested 
that we just buy them directly from MS at full retail price, which is not much 
more that the Techsoup price.

Right now, we should be good for a couple of months, but I want to prevent any 
similar issues in future, and even the regular jumping-through-hoops that I 
have been doing up until now to keen the current system running.

The first issue to resolving this and improving long-term reliability, is to 
decide how many simultaneous users we realistically need.  The rest will flow 
from that.

Regards

paul

Re: Residential GPON last mile for network engineers (Telus AS852 and others)

[Paul Nash]

I have a Bell Canada gig fibre connection.  My first attempt was to bridge 
their all-in-one box (disaster, unreliable as all hell), second was to set a 
bunch of rules for inbound traffic.  Apart from inbound access being *very* 
iffy, their device was s_l_o_w.

So I pulled the fibre GBIC, used a small switch to grab the correct VLAN and 
pointed that at a small Cisco box.  Way more flexible, faster and more reliable 
than Bell’s box.  DSLreports had all the info needed to get the correct VLANs

YMMV

> On Oct 13, 2020, at 9:56 PM, Eric Kuhnke  wrote:
> 
> Very interesting. Looks like the intention is to bypass the ONT entirely and 
> use a GPON ONT SFP in ones own choice of small home router. If the ISP wants 
> to do some weird TR069 provisioning or other stuff it could be seen as 
> interfering with the proper management of their network if you remove the CPE 
> entirely.
> 
> In an ideal world, personally I would be totally fine with keeping a telco 
> provided small ONT configured as a dumb L2 bridge, with one optical interface 
> single strand (SC/APC) going to the ISP, and 1000BaseT to my own router.
> 
> On Tue, Oct 13, 2020 at 6:51 PM Eric Dugas  wrote:
> I don't have any particular insights for Telus, but there is a huge thread 
> about bypassing Bell ONTs on DSLReports: 
> https://www.dslreports.com/forum/r32230041-Internet-Bypassing-the-HH3K-up-to-2-5Gbps-using-a-BCM57810S-NIC
> Cheers,
> Eric
> On Oct 13 2020, at 9:38 pm, Eric Kuhnke  wrote:
> With the growth of gigabit class single fiber GPON last mile services, I 
> imagine a number of people reading the list must have subscribed to such by 
> now.
> 
> Something that I have observed, and shared observations with a number of 
> colleagues, is that very often a person who works for ($someAS) lives in a 
> location where you are effectively singlehomed to ($someotherAS). Maybe you 
> bought your house before you got a job with your current employer, or maybe 
> the network you work for doesn't do residential last mile service at all. 
> Perhaps you work remotely for a regional sized entity that's a long distance 
> away from where you live.
> 
> Therefore necessitating a choice of service from whatever facilities based 
> consumer-facing ISP happens to service your home.
> 
> For example, in Seattle, a number of people discovered that they could keep 
> the Centurylink GPON ONT, and remove the centurylink-provided router/modem 
> combo device. Provided that they were able to configure their own router 
> (small vyatta, pfsense box, mikrotik, whatever) to speak a certain VLAN tag 
> on its WAN interface and be a normal PPPoE / DHCP client.
> 
> I'm sure there are a lot of people who prefer to run their own home router 
> and wifi devices, and not rely upon a ($big_residential_isp) provided 
> all-in-one router/nat/wifi box with opaque configuration parameters, or no 
> ability to change configuration at all.
> 
> Any insights as to what the configuration of the Telus AS852 GPON network 
> looks would be helpful. Or other observations in general on 
> technically-oriented persons who are doing similar with other ILECs.

Re: iOS 14 (Apple) DNS bits

[Paul Ebersman]

vom513> Observation: iOS 14 now seems to send 3 queries (up from 2) for
vom513> every socket connection to a name.  Whereas we've had A
vom513> +  for quite some time in many OSes - on iOS 14 we now
vom513> have A +  + HTTPS (type 65).
[...]
vom513> Question: iOS 14 now flags networks that it believes are
vom513> blocking encrypted DNS.  It puts a warning in Settings for the
vom513> wifi.

Apple has made a number of unilateral decisions about how their phones
should work (in search of some definition of privacy) that are likely to
cause headaches for enterprise and others using something other than
apple blessed tech to secure their users. The mac addr randomization is
going to be another headache for IT.

From an apple developer on another list, official docs from apple and
some other things to read.

Developer documentation:







WWDC video/transcript:



"Encrypted resolvers designated by domain owners"  based on;

Re: SRv6

[Paul Timmins]

On 9/21/20 6:16 PM, Randy Bush wrote:
yes, privacy is one aspect of security. and, as mpls vns are not 
private sans encryption, they are not secure.

randy


As my backyard is not surrounded by a cement enclosure with acoustic 
baffling and white noise generators inside, it's not really private 
property.

Re: BFD for routes learned trough Route-servers in IXPs

[Paul Timmins]

On 9/17/20 1:51 PM, Douglas Fischer wrote:

But 30 Seconds for an IXP? It does not make any sense!
Those packets are stealing CPU cycles of the Control Plane of any 
router in the LAN.


Especially given how some exchanges lock the mac address of 
participants. You could probably get away with ARP timeouts of a day or 
even just permanent with manual clearing when you see a peer go down.


-Paul

Re: SRv6

[Paul Timmins]

My backyard is private. It offers no privacy with its chain link fence 
against a major street.


On 9/16/20 4:38 PM, Randy Bush wrote:

Privacy != encryption.

cleartext == privacy * 0

cleartext * complexity == privacy * 0

randy

Re: FCC: rulemaking on STIR/SHAKEN and Caller ID Authentication

[Paul Timmins]

A *LOT* goes through at least one TDM transition (so you can kiss that 
identity header goodbye). None of the big names in long distance 
termination support STIR/SHAKEN. There's about 4-5 that will do 
STIR/SHAKEN outside of testbed connectivity (my employer is one). One 
big name is still using a self signed certificate to sign their 
STIR/SHAKEN calls, it'll expire in a couple weeks so they should figure 
life out quickly. I won't shame them here.


The lions share of intercarrier traffic won't go through SIP until the 
big ILECs are required to interconnect over SIP in reasonable and 
non-discriminatory ways. I'm not holding my breath.


(AT Wireless and Verizon Wireless hide behind their respective 
landline networks generally, and without SIP connectivity to those, you 
won't be getting green checkmark calls to people on the two largest 
wireless carriers outside of private testbed connectivity anytime soon)


https://authenticate.iconectiv.com/authorized-service-providers-authenticate 



-Paul

On 9/10/20 4:09 PM, Michael Thomas wrote:


On 9/10/20 9:49 AM, Sean Donelan wrote:


At this month's FCC rulemaking meeting, it will consider

https://www.fcc.gov/document/fcc-announces-tentative-agenda-september-open-meeting-6 



Promoting Caller ID Authentication to Combat Spoofed Robocalls – The 
Commission will consider a Report and Order that would continue its 
work to implement the TRACED Act and promote the deployment of caller 
ID authentication technology to combat spoofed robocalls.

(WC Docket No. 17-97)



So I have a question: what percentage of traffic in the US is really 
coming from the legacy PSTN? My understanding is that it's pretty low 
these days.


If that's true, it seems to me that this is a SIP problem, not an 
e.164 problem.


Mike

Re: Don Smith, RIP.

[Paul Ferguson]

I was informed of this earlier today... I’ve known Don for almost all of 
the 20+ years that he was engaged with this community. I am very sad to 
hear of his passing.


Among his many accomplishments, one of the things that always impressed 
me about Don was that -- in addition to being a good friend and 
colleague -- he was also no-nonsense expert technical voice of reason 
and sanity (also a CCIE) and an early volunteer SANS Daily Incident Handler.


He will be sorely missed.

- ferg


On 7/23/20 4:22 PM, Dobbins, Roland wrote:



It is with a heavy heart that I must relate the news that Don Smith, formerly 
of CenturyLink and more lately of Netscout Arbor, passed away in his sleep last 
night.

Don was a colleague, friend, and mentor to many; he was a mainstay of the 
operational community, and tirelessly worked to make the Internet safer and 
more resilient for us all.  His intellect, wit, and generosity of spirit were 
well-known to those who were privileged to have the opportunity to work with 
and learn from him.

Don’s contributions to the industry were manifold.  While we are all diminished 
by his loss, his legacy abides; and we can honor him by continuing to build 
upon that foundation, for the betterment of the Internet community as a whole.

Once Don’s family have established plans for his memorial, they will be posted 
here.

Roland Dobbins 





--
Paul Ferguson
Tacoma, WA  USA
Illegitimi non carborundum.

Disney+ contacts or geolocation ideas

[Paul Nash]

I’m looking for a technical contact at Disney regarding geo-location.  I have a 
client (apartment building) with a /24 (one IP per apartment).  We recently 
upgraded out Internet connection to give a much-needed speed boost.  Same 
connectivity provider, same IP addresses, just a bigger pipe.

Since then, a while bunch of people have been unable to get to Disney+, while 
some can.  Those that fail have existing D+ subscriptions, and get “error code 
73”, which allegedly relates to location and rights management.

The bandwidth provider has checked DNS, reverse DNS and contacted Disney, to no 
avail.  WhoIS shows it as being in Toronto, Canada.

Meanwhile, there is a lynch mob forming, and a scaffold being built in the 
parking lot :-).

Any pointers on how to find someone at Disney with clue, who will be able to 
tickle their geolocation databane.

The really irritating part is that everything worked until we had the bandwidth 
upgrade.

Re: 60ms cross continent

[Paul Nash]

Not quite VSAT, but in the bad old SA days (pre-demicracy), I did some work for 
a company that used a UK-based satellite provider for data to the client (data 
was sent in the VBI), and dial-up for the traffic from the client.

Still relied on a local provider for the dial-up, though, so could be censored.

Before TICSA, I also looked at buying a private (pirate) satellite earth 
station.  The Russian government were selling off surplus 8-wheel-drive 
military satellite earth stations, and I was thinking of parking one in my back 
garden (I lived on a farm).

paul

> On Jul 9, 2020, at 12:44 PM, Mark Tinka  wrote:
> 
> 
> 
> On 9/Jul/20 17:51, Joel M Snyder wrote:
> 
>> Oh man I wish that were wholly true... Satellite/VSAT has another very
>> very important attribute: it's not subject to the whims of the local
>> government or regulators.  So when there's an election or some unrest or
>> coup or the prime minister has very bad flatulence, and some person says
>> "turn off the Internet," your non-terrestrial connection is there so
>> that you can continue to do business.
> 
> Very true, except there are still a few countries that require a single
> operator to have all "gateway" access out of the country, even via
> satellite. So yes, install, for sure. But if someone does the rounds and
> catches an "unlicensed" installation, that could be interesting.
> 
> 
>> (Plus, there are also still many places outside of capital cities in the
>> world where the Internet is truly awful and if you want bits, you have
>> to bring your own)
> 
> I did mention that use-case, already, in a previous post.
> 
> Simple applications such as ATM's in remote locations is still quite
> typical.
> 
> Mark.

Re: 60ms cross continent

[Paul Nash]

When we started TICSA (Internet Africa/Verizon/whatever), we went with a 9600 
bps satellite link to New Jersey specifically because the SAT-2 fibre had just 
been installed and traffic was being moved off satellite.  The satellite folk 
were getting *very* nervous, and gave us a heavily discounted service provided 
we had a 5-year contract that specified that they service *had* to run over 
satellite.  Job insurance.

As our requirements grew, we added fibre connections.  Eventually the telco 
canceled the satellite connection as they were starting to focus on VSAT.

paul

> On Jul 8, 2020, at 3:05 AM, Mark Tinka  wrote:
> 
> 
> 
> On 7/Jul/20 21:58, Eric Kuhnke wrote:
>> Watching the growth of terrestrial fiber (and PTP microwave) networks
>> going inland from the west and east African coasts has been
>> interesting. There's a big old C-band earth station on the hill above
>> Freetown, Sierra Leone that was previously the capital's only link to
>> the outside world. Obsoleted for some years now thanks to the
>> submarine cable and landing station. I imagine they might keep things
>> live as a backup path with a small C-band transponder MHz commit and
>> SCPC modems linked to an earth station somewhere in Europe, but not
>> with very much capacity or monthly cost.
>> 
>> The landing station in Mogadishu had a similar effect.
> 
> The early years of submarine fibre in Africa always had satellite as a
> backup. In fact, many satellite companies that served Africa with
> Internet prior to submarine fibre were banking on subsea and terrestrial
> failures to remain relevant. It worked between 2009 - 2013, when
> terrestrial builds and operation had plenty of teething problems. Those
> companies have since either disappeared or moved their services over to
> fibre as well.
> 
> In that time, it has simply become impossible to have any backup
> capacity on satellite anymore. There is too much active fibre bandwidth
> being carried around and out of/into Africa for any satellite system to
> make sense. Rather, diversifying terrestrial and submarine capacity is
> the answer, and that is growing quite well.
> 
> Plenty of new cable systems that are launching this year, next year and
> the next 3 years. At the moment, one would say there is sufficient
> submarine capacity to keep the continent going in case of a major subsea
> cut (like we saw in January when both the WACS and SAT-3 cables got cut
> at the same time, and were out for over a month).
> 
> Satellite earth stations are not irrelevant, however. They still do get
> used to provide satellite-based TV services, and can also be used for
> media houses who need to hook up to their network to broadcast video
> when reporting in the region (even though uploading a raw file back home
> over the Internet is where the tech. has now gone).
> 
> Mark.
> 

Re: free collaborative tools for low BW and losy connections

[Paul Nash]

> Exactly. And there's no disconnect: usenet doesn't scale because each object 
> is copied to all core nodes rather than referenced, or copied-as-needed, or 
> other.  This design of distributed messaging platform will eventually break 
> as it grows.  

Usenet scales far more gracefully than the current web.

Each node sends content to a few downstream nodes.  This makes it easy to 
scale; there is no central mega-node that gets overwhelmed, connectivity is to 
a nearby upstream where there is a reasonabe amount of bandwidth. Last time I 
ran a server, the sender could filter based on newsgroup or message size, so 
avoid swamping links.  Content was mostly text.

It is possible to use offline transmission — certain groups dumped onto mag 
tape and mailed, get pulled in at the destination.  BTDT.

More demand = more client nodes which in turn distribute to other nodes, so 
each node does not need to talk to a large number of others.

We did this about 30 years ago in South Africa; Rhodes university brought in 
most groups, I brought in alt.*.  We each distributed to a select number of 
nodes, who distributed again.  Lather, rinse, repeat.  Usenet for the entire 
sub-continent (along with email) over 9600 bps dial-up circuits.

paul

Re: South Africa On Lockdown - Coronavirus - Update!

[Paul Nash]

Don’t hold your breath :-(.

> On Mar 24, 2020, at 4:55 PM, Mark Tinka  wrote:
> 
> 
> 
> On 24/Mar/20 22:48, Randy Bush wrote:
> 
>> almost all our cultures have gaps; but some worse than others.  we will
>> all learn lessons in the coming many months of plague.  i know an office
>> which lost key engineers last year because they would not let them work
>> remotely.  now the entire company is working remotely, and successfully.
> 
> The Coronavirus is amplifying and accelerating the new economy that is
> burgeoning at the borders.
> 
> With some luck, those that need to pay attention, are.
> 
> Mark.

Re: free collaborative tools for low BW and losy connections

[Paul Ebersman]

woody> UUCP kicks ass.

And scary as it sounds, UUCP over SLIP/PPP worked remarkably
robustly. When system/network resources are skinny or scarce, you get
really good at keeping things working.

:)

Re: South Africa On Lockdown - Coronavirus - Update!

[Paul WALL]

On Tue, Mar 24, 2020 at 6:22 AM Alexandre Petrescu <
alexandre.petre...@gmail.com> wrote:

>
>
> Mr. Morrow - where are you situated approximately?
>
>
He's a network operator. From North America, on the North American Network
Operators mailing list. Something you are not, so please stop spouting your
drivel on a list that has nothing to do with you. This is a crisis, not a
time for a European Project Proposer  to
spout off massively uninformed bullshit non-stop because no one else will
listen.

NANOG-L mods: it's time to show some leadership.

Re: DHS letters for fuel and facility access

[Paul Nash]

You just have to make sure that you test the right thing.

In a former life I was an electrical engineer. My first job was with a 
consulting engineering firm; out biggest customer was the biggest supermarket 
chain in South Africa.  One of my tasks was to travel to one of their stores 
each Saturday after closing (those were the days when they closed at noon on a 
Saturday until Monday morning) and test their stand generators.

The manager’s idea was usually to press the start button, check that the big 
diesel started, then shut down and go home.  My idea was to pull the main 
incoming breaker.  9 times out of 10 on first visit, the diesel would start, 
and then die as soon as the load kicked in because of carbon buildup in the 
cylinders.

After discussions with the supermarket management, they decided to (a) have all 
the diesels serviced ASAP, and (b) adopt my protocol of start diesel, wait for 
it to come under load, run for at least 30 minutes to get up to heat and clear 
the carbon deposits.

I use a similar technique for failover tests on servers, routers, firewalls — 
pull the power cord and see what happens, pull the incoming network and see 
what happens.

This was stymied by a recent network outage where the ISP network was up and 
running, connected back to their local PoP and thence to their backbone, but 
connectivity from that network to the critical servers was down.  So now we 
test end-to-end that the server is reachable, and let the network fail over if 
not.

paul

> On Mar 18, 2020, at 11:56 AM, Karl Auer  wrote:
> 
> An untested emergency system has to be regarded as a non-existent
> emergency system.
> 
> No matter how painful it is to test, no matter how expensive it is to
> test, the pain and the expense are nothing compared to the pain and
> expense of having an actual emergency and discovering that the
> emergency system doesn't work...
> 
> Multiplied by infinity if it costs lives.
> 
> Regards, K.
> 
> -- 
> ~~~
> Karl Auer (ka...@biplane.com.au)
> http://www.biplane.com.au/kauer
> http://twitter.com/kauer389
> 
> GPG fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
> Old fingerprint: 8D08 9CAA 649A AFEF E862 062A 2E97 42D4 A2A0 616D
> 
> 

Re: DHS letters for fuel and facility access

[Paul Nash]

September 2001.  Just after the 9/11 attacks, all of lower Manhattan was shut 
down.  Out link (IIRC) was to a satellite farm on Staten island, across the bay 
to 60 Hudson.  Power went off, diesels kicked in, fuel trucks was not allowed 
in, and a few days later we lost all international connectivity.

Lots of important people lost power as well, so the feds decided to let the 
diesel tankers in after a few days’ deliberations.

paul

> On Mar 17, 2020, at 11:21 AM, Mark Tinka  wrote:
> 
> 
> 
> On 17/Mar/20 17:15, Paul Nash wrote:
> 
>> That same fuel shortage killed all Internet traffic to sub-Saharan Africa.  
>> Took us a while to figure out what was wrong with the satellite link to the 
>> US.
> 
> What year was that :-)?
> 
> Mark.

Re: DHS letters for fuel and facility access

[Paul Nash]

That same fuel shortage killed all Internet traffic to sub-Saharan Africa.  
Took us a while to figure out what was wrong with the satellite link to the US.

paul

> On Mar 16, 2020, at 5:12 PM, Ben Cannon  wrote:
> 
> We (Verizon not me) lost a central office during 9/11 because it ran out of 
> fuel - the tankers were staged but we’re not allowed to enter Manhattan.  
> 
> This clears that pathway for us now, and it’s fairly standard protocol since.
> 
> -Ben
> 
>> On Mar 16, 2020, at 1:20 PM, Sean Donelan  wrote:
>> 
>> 
>> On some other mailing lists, FCC licensed operators are reporting they have 
>> received letters from the Department of Homeland Security authorizing 
>> "access" and "fuel" priority.
>> 
>> Occasionally, DHS issues these letters after natural disasters such as 
>> hurricanes for hospitals and critical facilities.  I haven't heard of them 
>> issued for pandemics.
>> 

Re: COVID-19 vs. our Networks

[Paul Nash]

> … as soon as they enter the Province
> from outside Canada they are "requested" to self-isolate for 14-days.
> This is for citizens.  Don't know what the policy is for non-Canadians.

Maybe not so much in practice.  I landed at Pearson late last night, returning 
from South Africa via Amsterdam.  Other than the standard passport checks, no 
sign of any screening.  Yay Doug Ford and his merry men.

I’m in self-quarantine for the next 14 days, working from home, in case of any 
symptoms.  I obviously hope the there are none, and that I can go back to visit 
clients, but I feel that that would be a really stupid thing to do right now.

In the meantime, schools are shut down, and I have two children back home from 
university.

paul


> 
>> (Fortunately, I'm in a position to hide in my apartment and only
> emerge
>> for grocery shopping at 2AM until things wind down... Hope everybody
> else
>> has a good contingency plan)
> 
> Yeah, sounds like a plan.
> 
> -- 
> The fact that there's a Highway to Hell but only a Stairway to Heaven
> says a lot about anticipated traffic volume.
> 
> 
> 

Re: QUIC traffic throttled on AT residential

[Paul Timmins]

It's okay though, because we freed up UDP/53 by moving DNS to TCP/443, 
so then we can move HTTPS to UDP/53.


On 2/21/20 6:37 PM, Owen DeLong wrote:

First we moved the entire internet to TCP/443.

Now we propose moving it all to UDP/53.

What’s next? Why not simply eliminate port numbers altogether in favor 
of a single 16-bit client-side unique session identifier.


Owen

On Feb 21, 2020, at 15:20 , Matthew Petach > wrote:




On Fri, Feb 21, 2020, 13:31 Łukasz Bromirski > wrote:



[...]

Now… once we are aware, the only question is — where we go from here?

—
./



Well, it's clear the UDP 443 experiment wasn't entirely successful.

So clearly, it's time to use the one UDP port that is allowed through 
at the top of everyone's ACL rules, and update QUIC in the next 
iteration to use UDP/53.


*THAT* should solve the whole problem, once and for all.

;)

Matt

Re: [EXTERNAL] Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Paul Ebersman]

gleduc> I remember that TI luggable - that sucker weighed a ton!

U of I used those in the libraries. I remember looking up books for
inter-library/lincoln trail and handing the printout to
students. Problem was that clay or whatever it was that made the paper
worked didn't last for more than a month or two before it faded to
illegible, as many grad students found out...

Microsoft mail delivery issue

[Paul Kelly - Blacknight]

Hi There,

If there are any Microsoft mail admins on the list can they please contact me 
ASAP. We’re having deliverability problems into you and all the usual tools 
don’t help with fixing problems, only diagnosing them after the fact.

Thanks,

Paul

Paul Kelly
CTO
Blacknight Internet Solutions Limited
Cloud Hosting, Colocation, Dedicated servers, IP Transit Services
ISO 27001:2013 Certified
Tel: +353(0)599183072
Lo-call: 1850 929 929
DDI: +353 (0) 59 9183091
Skype: flamegrill


e-mail: p...@blacknight.com<mailto:p...@blacknight.com>
web: http://www.blacknight.com


Blacknight Internet Solutions Ltd, Unit 12A,Barrowside Business Park, Sleaty 
Road, Graiguecullen, Carlow, Ireland. Company No.: 370845

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Paul Nash]

Carrying on with the “first Internet connection” thread:

I forget how I found out about Usenet and UUCP email (lost in the mosts of 
time).  I ran a store and forward dial-up link from South Africa to DDSW1 in 
Chicago (Hi Karl!  Thanks!).  I cobbled together a package with a DOS-based 
mail reader and a DOS port of UUCP that several people used to get their email 
(including a local medical research establishment and the local veterinary 
college).  Demand grew, along with a request to relay email to the UNHCR in 
Northen Mozambique, so I scraped some money together to import a horribly 
expensive Telebit modem.  I ended up being the regional non-academic email hub 
for Southern Africa.

Just prior to the 1994 election, I got together with a two friends (Alan Barret 
and Chris Pinkham) and founded the first ISP in sub-Saharan Africa.  We managed 
to get a 64k satellite link at a very good price (the satellite folk were busy 
being retrenched and we were prepared to sign a contract specifically requiring 
satellite service for 5 years, which gave them some job security).  We borrowed 
a Cisco router from DiData (Cisco agents), skirted other telco regulations to 
link regions.

One of our early customers was a group of students who wanted to start a small 
dial ISP nearby.  We gave them service, bootstrapping what became our biggest 
competitor, Internet Solutions (now part of DiData, who never did ask for their 
router back).  Our little ISP grew and grew, and eventually merged with our 
biggest client, was sold, sold again, and so on.  Last time I looked, it had 
become Verizon Africa.

paul

> On Jan 28, 2020, at 6:40 PM, Forrest Christian (List Account) 
>  wrote:
> 
> So to add my two stories:
> 
> I provided the Idea and a whole bunch of time/labor/etc to start a dialup ISP 
> in our hometown back in 1994.   I remember having a big debate on whether to 
> bring in a single 56K leased line or 128K fractional T1.  We went with the 
> Fractional T1 just because it could be easily expanded over time.   (That T1 
> is now multiple 10GB circuits - yes the ISP is still running and I still am 
> involved).   So a single 128K fractional T1, a cisco 2501 (with external DSU, 
> those internal cards didn't exist yet), and 8 14.4 modems attached to a 
> single Sun Unix box.  Note that this was pre-web, and back in the days where 
> you pretty much knew at least generally everything which was on the internet.
> 
> Things grew quickly, don't remember how many lines.   At some point we moved 
> to having 56K modems on our end, which required a digital carrier to the 
> central office.   T1's were very expensive, so we did a bit of tariff 
> arbitrage.   One could obtain a 'metered' ISDN BRI line for like next to 
> nothing - the metering had to do with the fact they were going to charge you 
> by the minute for any calls, but here's the catch:  for outgoing calls only, 
> incoming calls were free which worked great for a dialup ISP.The problem 
> was that 56K dialin concentrators all wanted T1 lines.What we discovered 
> is that Adtran made a box which would take a whole bunch of ISDN BRI (each 
> with 2 channels), and combine them into a single T1.   And due to the retail 
> pricing difference for T1 vs BRI, we could pay for the box in a few months.   
>  So we took a whole truckload of ISDN BRI lines and combined them into a few 
> channelized T1's and ended up paying a lot less to the phone company.
> 
> Of course, things have grown past that (we have an extensive WISP network and 
> have an ever-growing amount of fiber in the ground).  But it's fun to think 
> about where we started.
> 
> On Mon, Jan 27, 2020 at 1:00 PM  wrote:
> 
> On January 27, 2020 at 22:57 ma...@isc.org (Mark Andrews) wrote:
>  > The hardware support was 2B+D but you could definitely just use a single 
> B.   56k vs 64k depended on where you where is the world and which style of 
> ISDN the telco offered. 
> 
> FWIW bulk dial-up lines were often brought in as PRIs which were 24
> ISDN 2B+D lines on basically a T1 (1.544mbps) and then you could break
> those out to serial lines.
> 
> The sort of cool thing was that you could get caller information on
> those even if the caller thought they blocked it with *69 or whatever
> it was and log it. I forget the acronym...no no, that's the usual
> caller-id this was...u, DNI? Something like that.
> 
> I won a court case with that data.
> 
> -- 
> -Barry Shein
> 
> Software Tool & Die| b...@theworld.com | 
> http://www.TheWorld.com
> Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
> The World: Since 1989  | A Public Information Utility | *oo*
> 
> 
> -- 
> - Forrest

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Paul Ebersman]

wsimpson> When we first designed PPP in the late '80s to replace SLIP
wsimpson> and SLFP, it was expected to run at 300 bps and scale up, so
wsimpson> the timeouts reflected that.  When I designed PPP over ISDN,
wsimpson> added language to allow faster retransmission.

SLIP and PPP were quite... robust. Some UCB folks managed to get SLIP
over tin can and string. Two acoustic coupler 150b modems, 2 8oz V8 cans
and waxed cotton thread.

wsimpson> Like many of you, I started an ISP in 1994 with a 56 kbps
wsimpson> uplink, and only 6 local customers  The routers were in a
wsimpson> bathroom over the garage.

Our first CA hub was in the janitor's closet at a now defunct computer
company. We initially had problems with the janitors unplugging the
router on weekends to plug in their floor buffers.

Ah, the good old days.

Re: akamai yesterday - what in the world was that

[Paul Nash]

> first personal connection was a dedicated dialin using a telebit
> trailblazer at 9600 bps. that was a benefit of work.

The Telebits were awesome over impaired lines.  Their funky modulation scheme 
let them get through where nothing else would (like using barbed wire fences 
instead of phone wire).

I used them to link up the UNHCR in Northern Mozambique.  Only problems were 
when someone opened a gate in the fence to move cattle — no carried until they 
closed it again.

    paul

Re: akamai yesterday - what in the world was that

[Paul Ebersman]

first internet for me was a 300 baud modem from offsite to someplace
buried in the pentagon that I think aggregated all of us into a single
56k upstream.

at 300 baud, you could actually read faster than the screen scrolled. we
started getting 1200 baud, then 2400 baud but the USAF wouldn't let you
upgrade just to get "faster", only to replace broken gear... so a large
number of 1200 baud modems somehow dropped accidentally on the
floor. 10-15 times in some cases.

first personal connection was a dedicated dialin using a telebit
trailblazer at 9600 bps. that was a benefit of work.

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Paul Ebersman]

kauer> When *I* were a lad we had to touch the wires with our tongues to
kauer> tell one from zero, no job for a sissy lemme tell you.

Wires? You had wires? We had to cut out our own intestines, braid them
into strands and dip them in salt water to make them conductive.

Our bosses would feed us a cup of sulphuric acid, work us in 29 hour
shifts, then kill us, and dance over our graves, singing Hallelujah.

kauer> You tell that to young folk these days and they don't believe
kauer> you...

Nope, Nope.

Damn millenials... Killing the internet...

Re: akamai yesterday - what in the world was that

[Paul Nash]

> So, I grew up in South Africa, and one of the more fascinating /
> cooler things I saw was a modem which would get you ~50bps (bps, not
> Kbps) over a single strand of barbed wire -- you'd hammer a largish
> nail into the ground, and clip one alligator[0] clip onto that, and
> another alligator clip onto the barbed wire. Repeat the process on the
> other side (up to ~5km away), plug the modems in, and bits would
> flow... I only saw these used a few times, but always thought they
> were cool….

Do you remember anything about the actual type of modem?  Or where you deployed 
them?

In the days before the Internet came to SA, I ran a dial-up email link between 
the US and Pretoria, polled by various people locally (including CSIR, SAIMR).  
I also carried mail for the UNHCR in Northern Mozambique.  Mail came via Karl 
Deninger (DDSW1) in Chicago, IIRC.

They were missing several kilometres of phone wire, so connected the link to 
the fence on each side of the road.  We get about 1200bps on a good day IIRC, 
and would loose carrier whenever someone moved cattle from one field to another 
and opened a gate in the fence.

paul

Re: akamai yesterday - what in the world was that

[Paul Ebersman]

bzs> When we, The World, first began allowing the general public onto
bzs> the internet in October 1989 we actually had a (mildly shared*) T1
bzs> (1.544mbps) UUNET link. So not so bad for the time. Dial-up
bzs> customers shared a handful of 2400bps modems, we still have them.

The World was also our (UUNET) Boston hub. And at that time,
cross-country core backbone links were T1. We all thought the NSF T3
backbone was a government boon-doggle. :)

Re: akamai yesterday - what in the world was that

[Paul Nash]

> I find it both happy and disturbing.  I remember the first 2.4/2.5g links I 
> turned up as well as the first 10g and (eventually) the first 100g links.
> 
> I was leaving the house earlier this week thinking about how it used to be 
> Mbps of traffic that was a lot and now it’s Gbps and how that’s shifted to 
> Tbps.
> 
> While it makes me feel old, it’s also something that I marvel about 
> periodically.

A bit of perspective on bandwidth and feeling old.  The first non-academic 
connection from Africa (Usenet and Email, pre-Internet) ran at about 9600 bps 
over a Telebit Trailblazer in my living room.

The first non-academic IP connection was a satellite connection (64Kbps IIRC, 
not in my living room :-)).

Now we have a bajillion Gbps over submarine fibre landing pretty much 
everywhere, and my guess is that it is not enough bandwidth.

All this to bring such vital resources as Facebook and Netflix :-)

paul

Re: 5G roadblock: labor

[Paul Nash]

> 
> There are some wi-fi vendors who I know (and am currently testing) that
> have developed very cool centralized management tools for their wi-fi
> AP's, that include very interesting AI logic. It is pricier than a
> simple standalone enterprise-grade AP, or an AP you'll get from down the
> store. But it's still way cheaper than dense 5G deployment.

Depending on what you are after, folk like Ruckus and Cisco have had 
centrally-managed enterprise WiFi for many years.  I manage a Ruckus 
installation for an apartment building where there is one SSID from about 150 
APs, users have a unique password per apartment, which lands them onto that 
apartment’s VLAN, regardless of where they are in the building.

Works really well. 

I have seen Ruckus installations like this on university campuses, where users 
get access to different VLANs depending on who they are (but all use the same 
SSID).  Cisco have also been doing this for a long, long time (at far higher 
cost).

Not sure about Cisco, but the Ruckus stuff is also used widely in hotels and 
caravan parks where folk can buy a “day pass” — a shareable password that is 
valid for a pre-determined amount of time and will get them onto the wifi 
anywhere in the facility.  I’ve mostly seen Cisco in hospitals and banks.

In theory this could easily be spread through an entire suburb using outdoor 
APs.

paul

Re: 5G roadblock: labor

[Paul Nash]

> And more interestingly, if that city's residents and visitors had the
> option of connecting to active 5G or wi-fi, what do we think they'd choose?

They’d probably choose whichever popped un onto the device first.

FWIW, Rogers in Canada are moving to unlimited cellular data, with a monthly 
threshold, beyond which they reserve the right to throttle (but do not always 
throttle).  Bell probably do something similar.

The threshold increases with the number of devices on the account, and any 
throttling applies to all devices on that account.

    paul

Re: Iran cuts 95% of Internet traffic

[Paul Nash]

This was (not quite) how bits of sub-saharan Africa got netnews in the early 
days.  Store-and-forward, UUCP links over dial-ups, and the occasional mag tape 
couriered over.

paul

> On Dec 29, 2019, at 9:11 AM, Rich Kulawiec  wrote:
> 
> 
> And this is why, despite all the disdainful remarks labeling such
> things as "antiquated", mailing lists and Usenet newsgroups are vastly
> superior to web sites/message boards/et.al. when it comes to facilitating
> many-to-many communications between people.  Why?  Well, there are many
> reasons, but one of the applicable ones in this use case is that their
> queues can be written to media, physically transported in/out, and then
> injected either into an internal or external network seamlessly modulo the
> time delay.  And because the computing resources required to handle this
> are in any laptop or desktop made in the last decade, probably earlier.
> 
> If you're trying to get information in/out of a society that is raising
> network barriers to realtime communication, then you need methods that
> don't rely on a network and aren't realtime.
> 
> ---rsk
> 

Re: FCC proposes $10 Million fine for spoofed robocalls

[Paul Timmins]

On 12/20/19 9:00 AM, Mike Hammett wrote:

I can't imagine many telcos are making a lot of money from voice anymore.


We are. Not as much as the olden days, but we are. And a lot of 
companies charge surcharges to customers who have tons of short duration 
calls. Do the math on why, and who they're targeting for a little extra 
income.

Re: FCC proposes $10 Million fine for spoofed robocalls

[Paul Timmins]

On 12/19/19 6:11 PM, b...@theworld.com wrote:

They should be fining the telcos, they're making a lot of money on
these calls.

And if you believe otherwise (e.g., that it's like email spam) you've
been duped by telco PR.

Unlike spam when was the last time a telco failed to bill you for a
billable phone call? Never.

They know exactly who is using their system. And they get paid for
it. And these junk callers are making millions of calls per hour when
they're active.


I work for a phone company in a senior role, and have for years. I've 
also been saying this for years. These are all half solutions. The 
people handling these calls know exactly who their customers are, and 
they'd remove them in hours if a legal mandate came down to provide 
passthrough penalties for providing service to these people. Legitimate 
callcenters don't send out tons of traffic with random source numbers 
that typically match the same first 6 digits as their caller.


It'd take the large and small companies alike maybe a day to run 
database queries to identify and shut down the callers doing this. But 
there's money to be made in prolonging the issue - they get to charge 
the caller for making the calls, and the customers to block them.


(for what it's worth, the problem ones aren't on my network. I checked.)

-Paul

RE: [EXTERNAL] RE: DDoS attack

[Paul Amaral via NANOG]

Rarely will sourced ips be the same every time a victim gets DDOS'd. Good 
telemetry is key but every time the attack happens it needs to be looked at.  I 
find bogon prefixes are not as used much, especially amplification attacks.  
Gathering good intel and blocking bogons will help,  but there is no one 
strategy that works. You also will always risk blocking some good traffic. 
Again, there's a reason why you can only mitigate and not stop a DDOS 
completely. 

Paul  

-Original Message-
From: Nikos Leontsinis  
Sent: Tuesday, December 10, 2019 5:19 PM
To: Aaron Gould ; 'Paul Amaral' ; 
ahmed.dala...@hrins.net; Nanog@nanog.org
Subject: RE: [EXTERNAL] RE: DDoS attack 

You can get the bogon prefixes from Cymru and defend your network using them in 
combination with rpf The key with the attacks dos or ddos is to have proper 
telemetry (streaming telemetry not polling telemetry) and baselines without 
this information you run the danger of blocking good traffic.

Based on the thread below I don't see any evidence of an attack only 
speculations.

nikos

-Original Message-
From: NANOG  On Behalf Of Aaron Gould
Sent: Tuesday, December 10, 2019 5:05 PM
To: 'Paul Amaral' ; ahmed.dala...@hrins.net; Nanog@nanog.org
Subject: [EXTERNAL] RE: DDoS attack

Years ago, we looked at netflow data and precursors to attacks, and found that 
UDP 3074 Xbox Live was showing up just prior to the attacks...and through other 
research we concluded that gamers are a big cause of large ddos attacks 
apparently they go after each other in retaliation

I've crafted a series of things for dealing with the results of volumetric ddos 
attacks... I've had attacks in upwards of 50 or 60 gig as I recall across 
all of my (3) internet connections at times

- deny acl's ... for ports/protocols that I know are absolutely not needed
- policers of various well known port attack vectors (gleaned from netflow data)
- policers of well-known *good* ports/protocols (like ntp, dns, etc) to some 
realistic level
- a repeat-victims list of ip's with policing udp for this group (note1)
- rtbh (note2)

Note 1 - Also, I've learned that if a customer has been attack once, the 
chances of them being the target of an attack again is highso by crafting 
the repeat victims list, you can catch next-day attacks of differing vectors.
Note 2 - for sustained attacks lasting a long time (30 mins, an hour, etc), we 
trigger a bgp/community route that goes out to the inet cloud and stops attack 
further into the upstream providers network... I know I "complete" the attack, 
but, I save my network ;) ...I use an old cisco 2600 as my trigger router and 
wrote a job aid that I shared with the NOC for triggering rtbh when needed, 
couple commands.
...I would like to automate my rtbh using what I understand is a possibly use 
case for FastNetMon, but haven't got around to it

I also wonder if team cymru's utrs project and other things like that would 
benefit my security posture.


-Aaron


This email is from Equinix (EMEA) B.V. or one of its associated companies in 
the territory from where this email has been sent. This email, and any files 
transmitted with it, contains information which is confidential, is solely for 
the use of the intended recipient and may be legally privileged. If you have 
received this email in error, please notify the sender and delete this email 
immediately. Equinix (EMEA) B.V.. Registered Office: Amstelplein 1, 1096 HA 
Amsterdam, The Netherlands. Registered in The Netherlands No. 57577889.

RE: DDoS attack

[Paul Amaral via NANOG]

Normally these attacks are spoofed IPs, usually amplification attacks based on 
UDP using DNS/LDAP etc. This is something that is common and usually is towards 
schools, financial institutions. This an easy attack to orchestrate by anyone, 
most of these attacks can be launch via stresser services online. 800mbs to 
most smaller ISPs is a lot of traffic and can deeply impact not only the victim 
prefix but other non-targeted customers, as traffic consumed by the attack will 
cause problems for all users on that circuit.

There's a few things you can do, ask your upstream provider to rate limit UDP 
packets towards you. Rate limit them to what you think a normal UDP rate should 
be. I don’t recommend blocking UDP as you will block legit UDP packets from 
reaching any of your customer when the attack is not ongoing. Note most larger 
providers will not help or care to help, I know Comcast probably will not help 
you, their support techs will have no idea what you are taking about neither 
will most entry level engineers. However, it's worth taking a shot and asking 
you upstream provider. 

Another way you can minimize this is if you are multi-hommed with BGP. In this 
case take the targeted prefix and advertise to be preferred through one of your 
upstreams and move all over prefixes to the other link. This will ensure that 
most of your customers will not be impacted during the DDOS. Once you have the 
victim prefix preferred on that specific BGP link then you can rate limit on 
your edge, or the provider can do this for you. You will still have the full 
force of the attack at the edge unless you can get one of your providers to 
help you out. With DDOS you can only mitigate it and not necessarily stop it.  
Someone will always get that DDOS traffic. rather is your, your provider or 
your customers. The problem is figuring out where you want the traffic to be 
rate-limited, stopped etc and that who's expense. 

BTW those stresser services are usually free for a set about 0-15 min than you 
must pay thus why its not ongoing. 


Good luck, 

Paul 



-Original Message-
From: NANOG  On Behalf Of ahmed.dala...@hrins.net
Sent: Monday, December 09, 2019 3:08 PM
To: nanog@nanog.org
Subject: DDoS attack 

Dear All, 

My network is being flooded with UDP packets, Denial of Service attack, soucing 
from Cloud flare and Google IP Addresses, with 200-300 mbps minimum traffic, 
the destination in my network are IP prefixes that is currnetly not used but 
still getting traffic with high volume.
The traffic is being generated with high intervals between 10-30 Minutes for 
each time, maxing to 800 mbps When reached out cloudflare support, they 
mentioned that there services are running on Nat so they can’t pin out which 
server is attacking based on ip address alone, as a single IP has more than 
5000 server behind it, providing 1 source IP and UDP source port, didn’t help 
either Any suggestions?

Regards,
Ahmed Dala Ali 

Re: Equinix

[Paul Zugnoni via NANOG]

I'll second Martijn's comment and add this: Never choose "Next Available."
It's the easy route up front but painful the rest of its life. We started
predetermining where we wanted each of our xconnects (regardless which colo
company) and submitting the port numbers with tickets / Equinix
portal-based request. And it's been way smoother sailing since. You are you
own source of truth, not the tech that serviced your ticket.  PZ

On Thu, Dec 5, 2019 at 8:33 AM Grzegorz Janoszka 
wrote:

> On 05/12/2019 17:10, Martijn Schmidt via NANOG wrote:
> > You're probably best off ordering those crossconnects through the
> > Equinix portal, then you can choose the exact positions for the order
> > that goes to the facility rather than relying on a human to transcribe
> > them correctly from your PDF.
>
> If only Equinix portal reflected how your patch panels really look like...
>
> --
> Grzegorz Janoszka
>


-- 
PZ
Head of Datacenter and Network Infrastructure, Wish
p...@wish.com +1-650-313-3458

Any Charter email admins monitoring this list?

[Paul Gover]

I'm in search of someone from Charter who can help get a small ISP's 
email servers off the blacklist for stny.rr.com. If you could reach out 
off list, it would be much appreciated.  (Email to 
priorityescalationt...@charter.com was rejected as undeliverable.)


Kind regards,
Paul Gover, Adams Cable Service