Re: DMARC ViolationAS21299 - 46.42.196.0/24 ASN prepending 255 times

[Baldur Norddahl]

On Sun, 27 Mar 2022 at 18:31, Jon Lewis  wrote:

> Is prepending used for any purpose other than TE?  The point I think Joe
> was trying to make was prepending once or even a few times has uses.
> Prepending more than a few times is unlikely to accomplish anything a few
> prepends didn't get done.
>

I suppose so-called "backup routes" could also be called traffic
engineering yet it is different from the use case I described.

I understand the "diameter of the internet" to mean the maximum number of
unique AS numbers in an AS PATH observed in any route in my DFZ routing
table. Say I have two IP transit uplinks and I want one to be strictly
backup meaning I want to receive no traffic unless the other is down. I
might then prepend at least "the diameter of the internet" and that would
be enough. Any more prepends will do nothing. This could probably be proven
mathematically for the worst case, although in reality you would not even
need that many prepends to get the effect.

However using prepends for traffic engineering in the sense prioritizing my
peers relatively to each other is completely different. Especially true
when some are peers on internet exchanges (not IP transit). Here the
diameter of the internet is completely irrelevant. What matters is the
number of classes I can make up for my peers. I admit those two numbers
might not be all that different, but I feel it is still worth pointing out
the error in the logic.

The logic is wrong even for the backup case. Say I have an extreme of N x
IP transits and I want all of them to be backups in a strict order. Such
that all traffic comes in on transit A. If transit A is down, then
everything should use B. If A and B are down then 100% to C etc. In that
case I would need to prepend "the diameter of the internet" on B and "the
diameter of the internet" times two on C etc. Why times two and not + 1?
Because when A is down we have B with a number of prepends. C needs to have
"the diameter of the internet" more than B to be sure no traffic goes that
way when B is active.

Prepending 50, 100, 200+ times is kind of a universal "We have no clue
> what we're doing and you should reject our routes."
>

That is likely yes.

Regards,

Baldur

Re: DMARC ViolationAS21299 - 46.42.196.0/24 ASN prepending 255 times

[Baldur Norddahl]

On Fri, 25 Mar 2022 at 17:32, Joe Provo  wrote:

> That said, prepending pretty much anything more than your current view
> of the Internet's diameter in ASNs is useless in practice.
>

That is one way of viewing it. But prepending can also be used for traffic
engineering. I could prepend 1 to my free peers, 2 to my paid peers, 3 to
cheap ip transit, 4 to expensive ip transit etc. The linked draft RFC does
not appear to discuss this at all. The depth of prepending used this way
only relates to how many different classes of peers you can imagine in your
setup and is not at all related to the "internet's diameter".

To someone on the other side of the planet, who are neither peers nor
customers of peers, they will just observe that I am prepending 3 or 4
times and wondering why the extra prepends? The answer is that closer to my
home there are people who are observing the same routes with 1 or 2
prepends and that it matters.

The draft RFC lists some alternatives to prepends of which none can do
anything of the sort I just described.

Regards,

Baldur

Re: questions about ARIN ipv6 allocation

[Baldur Norddahl]

On Mon, 6 Dec 2021 at 19:08, Owen DeLong via NANOG  wrote:

>
> Unfortunately, when the board did change the terms, it was made quite
> clear that the only way to terminate the LRSA was to surrender my resources
> in the process.
>

You could transfer the resources to RIPE... :-)

Re: private 5G networks?

[Baldur Norddahl]

On Tue, 30 Nov 2021 at 23:48, Shane Ronan  wrote:

> Please provide details on public transit systems that are controlled via
> Wifi, I find that very interesting.
>

This should give a good overview:

https://backend.orbit.dtu.dk/ws/files/128950142/COMST2661384.pdf

It is in fact quite interesting.

And yes these are low bandwidth but on the other hand often stretch wifi to
the very limits on the distance between bases. I am not claiming this is
the same use case as a warehouse. I am pointing out that the argument that
a system critical implementation _must_ be based on licensed frequencies
does not hold as nothing could be more critical than a system that prevents
trains from colliding.

I do claim that the reason these metro train systems can boast of a very
high uptime is not that it would be especially hard to jam their wifi based
systems. No it is in fact probably quite easy to do so. It is just that
nobody does it. Because that way lies jail and there are also so many other
ways to stop the trains (rocks on the tracks etc). The same holds true for
the warehouse as someone trying to cause trouble could just as easily do
something to the power, cut a fiber cable, start a fire, call in a bomb
threat, etc.

Also having a licensed frequency only stops those that are law abiding and
it is never legal to cause harmful interference to sabotage the operations
of a warehouse.

That leaves the risk that the wifi frequencies are blocked by other legal
users of the frequencies. This risk is especially low on the new 6 GHz
frequencies because the range is not great and you do have full control of
what equipment enters your warehouse. The risk is essentially that the
neighbor is also a warehouse with a wifi based system. The physical
separation would in most cases be enough that this is not a problem and
otherwise it would not be too much trouble to talk to the neighbor to agree
on some frequency split on the bases at the border between the two systems.
No need to pay a third party or the government for that.

I did read about a use case for a private 5G network however. A system
covering the harbor. Wifi would be at a disadvantage here because it is a
large outside area with a lot of third parties entering, both ships and
trucks. I imagine there also exists similar such a large mining operation
etc.

Regards,

Baldur

Re: private 5G networks?

[Baldur Norddahl]

tir. 30. nov. 2021 23.19 skrev Tom Beecher :

> In my view there is no practical difference. The owner has full control of
>> his warehouse and it would be very illegal for any outside party to install
>> any device at all including unauthorised wifi devices.
>>
>
>  Nothing illegal about someone sitting in a parking lot next door with a
> pineapple turned up to 11 that's washing out all the normal wifi spectrum.
>

If we are talking about wifi 6E on 6 GHz sitting in a parking lot trying to
cause harmful interference within legal limits will not successfully harm
the operation within a building, especially not if the owner has a security
perimeter. Harmful interference on purpose is not legal in any case.


> It would be illegal to do that with CBRS.
>

On the other hand, saboteurs rarely care about legal and can easily jam
either system.

And yet, this is simply not a real problem. Did you know that a larger
number of train transit systems are controlled by WiFi? Block that WiFi
signal and the trains stop city wide. But has this ever happened?

Regards

Baldur

>

Re: private 5G networks?

[Baldur Norddahl]

tir. 30. nov. 2021 22.09 skrev Shane Ronan :

> Happy, no, but it wouldn't be illegal. And if they are building their
> warehouse automation based on wifi, it would surely be a problem if someone
> was competing for bandwidth.
>

In my view there is no practical difference. The owner has full control of
his warehouse and it would be very illegal for any outside party to install
any device at all including unauthorised wifi devices.

For comparison, consider that many city train systems are operating
signaling using wifi equipment.

Regards

Baldur

Re: IPv6 and CDN's

[Baldur Norddahl]

man. 29. nov. 2021 02.12 skrev Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp>:

>
>
> > The only way to truly reduce Opex at scale is automation.
>
> Automation by what? DNS?
>
> Masataka Ohta
>


Most of our customers are provisioned by Radius. The remaining are
configured by scripting using Netconf.

We use DNS to document the network. If our DNS was down and I need to
connect to a router in some city, do you really expect me to remember the
IP address? I would have to look it up and our chosen database for that
happens to be DNS. It has some obvious advantages.

Regards

Baldur

>

Re: IPv6 and CDN's

[Baldur Norddahl]

søn. 28. nov. 2021 13.59 skrev Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp>:

>
> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.
>

99% if not 100% of our subnets have either only routers or only hosts + a
gateway. So that would be a strange rule to follow. Also very expensive if
we are talking public addressing.

I find that 10.x.y.z is not much if you want to have a system in your
subnet numbering. With ipv6 there is much more space to enable systematic
numbering schemes.

>

Re: multihoming

[Baldur Norddahl]

On Wed, 24 Nov 2021 at 08:16, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> So, as modifying end systems is inevitable, there is
> no reason not to support full end to end multihoming
> including modifications to support multiple addresses
> by TCP and some applications.
>
> Masataka Ohta
>

Are you proposing SCTP? There is sadly not much more hope for widespread
adoption of that as of IPv6.

Regards,

Baldur

Re: Anyone else getting the 'spam' bomb threat?

[Baldur Norddahl]

On Tue, 19 Oct 2021 at 19:20, Kain, Becki (.)  wrote:

> The thing is, who is in office to care?  Oh wait, guess equipment *is*
> important
>
>
For how long did you keep up with the evacuation of the equipment? :-)

Re: DNS pulling BGP routes?

[Baldur Norddahl]

On Mon, 18 Oct 2021 at 09:51, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> But, with settlement free peering between tier 1 ISPs, tier 2
> ISPs having transit/paid peering with a tier 1 ISP will receive
> routes from peers of the tier 1 ISP. There is transit traffic
> exchanged between tier 1 ISPs over settlement free peering.
>
> So, I don't think distinguishing transit from peering
> meaningful for precise discussions.
>

Around here there are certain expectations if you sell a product called IP
Transit and other expectations if you call the product paid peering. The
latter is not providing the whole internet and is cheaper.

The so-called "tier" of a company is a meaningless term. Traffic will never
traverse two settlement free peering links and this is true for "tier 1"
ISPs as well. Paid peering is understood to be the same as a settlement
free peering except for not being settlement free. Therefore a paid peering
with an "tier 1" ISP will not provide any traffic that traverses their
settlement free peering links with other "tier 1" ISPs. It is quite
possible some "tier 1" ISPs do not see the point in providing such a
product but then they just won't offer paid peering - only IP transit.

In more technical terms, no peering link, settlement free or for pay, has
routes for the whole internet. If the peering had routes for the whole
internet it would be IP transit. This is achieved by only announcing own
customer routes on the peering links and _not_ announcing routes received
from other peering links. You get access to their customers but you need to
make other arrangements to get access to the rest of the internet.


>
> > For smaller ISPs it works the other way around. An evil CDN could
> > attempt to charge us, the small ISP. I am happy that is not
> > happening.
>
> Because of natural monopoly and PON, most access/retail ISPs
> enjoy their domination in their own area regardless of their
> sizes.
>

This is not true in our part of the world. The regulator is requiring all
major last mile infrastructure owners to give access to reseller ISPs
breaking that monopoly. My own company both owns infrastructure (FTTH and
FTTB / apartment networks) and resell using FTTH / DSL owned by other
companies. Plus we have three 5G networks providing an alternative and also
breaking the monopoly.

Regards,

Baldur

Re: DNS pulling BGP routes?

[Baldur Norddahl]

søn. 17. okt. 2021 11.16 skrev Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp>:

> Jay Hennigan wrote:
>
> >> Access/retail ISPs have no problem by peering with neutral
> >> backbone providers.
> >
> > Neutral backbone providers don't peer with access/retail ISPs. They
> > sell transit to them.
>
> FYI, that is called paid peering.
>

Paid peering is not the same product as IP Transit. In general a packet
never traverse two peering links because that would mean the middle man is
not getting paid to move the traffic. Paid peering with a backbone provider
will get you routes from their paying customers but not from their peers.
The same as you would have from a settlement free peering.



> >> CDN provided backbone only reduces costs of other backbone
> >> providers without reducing costs of access/retail ISPs.
> >
> > Access/retail ISPs that peer with CDNs eliminate the cost of paying
> > for transit for the content delivered by the CDN. That's what the
> > initials CDN stand for.
>
> But, it does not mean both parties of the peer are equally
> benefited. As such peering may be paid one, though it
> may not be the current practice.
>
> Given the observed profitability of CDN providers, CDN
> providers are, seemingly, more benefited (because they
> are not neutral), in which case, CDN providers should
> pay to access/retail ISPs.
>
> Masataka Ohta
>

I do not want Netflix to pay me. I get paid by my customers, some of which
also happens to be Netflix customers. If Netflix had to pay me, they would
need to get that money from the same people who are already paying me
directly. What is the point of that?

Let me tell you the point. Large ISP can exploit their domination of the
marked to double dip, which means they want to be paid twice. That happens
to be not neutral and is a way to make the customer pay a hidden fee.

For smaller ISPs it works the other way around. An evil CDN could attempt
to charge us, the small ISP. I am happy that is not happening.

Regards

Baldur

Re: facebook outage

[Baldur Norddahl]

man. 4. okt. 2021 23.33 skrev Bill Woodcock :

>
>
> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock  wrote:
> >
> >
> >
> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock  wrote:
> >>
> >> They’re starting to pick themselves back up off the floor in the last
> two or three minutes.  A few answers getting out.  I imagine it’ll take a
> while before things stabilize, though.
> >
> > nd we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and
> fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some third-degree
> burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives, so this
> doesn’t happen again.  :-)
>


We have had dns back for a while here but the site is still down. Not
counting this as over yet.

Re: massive facebook outage presently

[Baldur Norddahl]

Not in such a primitive fashion no. But they could definitely have a
secondary network that will continue to work even if something goes wrong
with the primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli  wrote:

> Seems unlikely that FB internal controls would allow such a backdoor ...
>
> "Never to get lost, is not living" - Rebecca Solnit
>
> Sent with ProtonMail <https://protonmail.com/> Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
> baldur.nordd...@gmail.com> wrote:
>
>
>
> On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:
>
>>
>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>
>>
>> I believe the original change was 'automatic' (as in configuration done
>> via a web interface). However, now that connection to the outside world is
>> down, remote access to those tools don't exist anymore, so the emergency
>> procedure is to gain physical access to the peering routers and do all the
>> configuration locally.
>>
>> Assuming that this is what actually happened, what should fb have done
>> different (beyond the obvious of not screwing up the immediate issue)? This
>> seems like it's a single point of failure. Should all of the BGP speakers
>> have been dual homed or something like that? Or should they not have been
>> mixing ops and production networks? Sorry if this sounds dumb.
>>
>
> Facebook is a huge network. It is doubtful that what is going on is this
> simple. So I will make no guesses to what Facebook is or should be doing.
>
> However the traditional way for us small timers is to have a backdoor
> using someone else's network. Nowadays this could be a simple 4/5G router
> with a VPN, to a terminal server that allows the operator to configure the
> equipment through the monitor port even when the config is completely
> destroyed.
>
> Regards,
>
> Baldur
>
>
>
>
>
>

Re: massive facebook outage presently

[Baldur Norddahl]

On Mon, 4 Oct 2021 at 21:58, Michael Thomas  wrote:

>
> On 10/4/21 11:48 AM, Luke Guillory wrote:
>
>
> I believe the original change was 'automatic' (as in configuration done
> via a web interface). However, now that connection to the outside world is
> down, remote access to those tools don't exist anymore, so the emergency
> procedure is to gain physical access to the peering routers and do all the
> configuration locally.
>
> Assuming that this is what actually happened, what should fb have done
> different (beyond the obvious of not screwing up the immediate issue)? This
> seems like it's a single point of failure. Should all of the BGP speakers
> have been dual homed or something like that? Or should they not have been
> mixing ops and production networks? Sorry if this sounds dumb.
>

Facebook is a huge network. It is doubtful that what is going on is this
simple. So I will make no guesses to what Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using
someone else's network. Nowadays this could be a simple 4/5G router with a
VPN, to a terminal server that allows the operator to configure the
equipment through the monitor port even when the config is completely
destroyed.

Regards,

Baldur

Re: massive facebook outage presently

[Baldur Norddahl]

I got a mail that Facebook was leaving NLIX. Maybe someone botched the
script so they took down all BGP sessions instead of just NLIX and now they
can't access the equipment to put it back... :-)


man. 4. okt. 2021 20.31 skrev Billy Croan :

> I know what this is.  They forgot to update the credit card on their
> godaddy account and the domain lapsed.  I guess it will be facebook.info
> when they get it back online.  The post mortem should be an interesting
> read.
>
> On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl 
> wrote:
>
>> Looks like they run there own nameservers and I see the soa records are
>> even missing.
>>
>> On Mon, Oct 4, 2021, 12:23 PM Mel Beckman  wrote:
>>
>>> Here’s a screenshot:
>>>
>>>
>>>
>>>  -mel beckman
>>>
>>> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke  wrote:
>>>
>>> 
>>> https://downdetector.com/status/facebook/
>>>
>>> Normally not worth mentioning random $service having an outage here, but
>>> this will undoubtedly generate a large volume of customer service calls.
>>>
>>> Appears to be failure in DNS resolution.
>>>
>>>

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Wed, 29 Sept 2021 at 22:11, Victor Kuarsingh  wrote:

> In the consumer world (Where a consumer has no idea who we are, what IP is
> and the Internet is a wireless thing they attach to).
>
> I am only considering one router (consumer level stuff).  Here is my
> example:
>

I am afraid you are tailor making your case. We could just as well have an
even more clueless customer that simply buys a 4G/5G router and attaches it
to the inside of his LAN in addition to the wifi router he got from his
DSL/cable/xPON service. Guess what will happen? It wont work as far as IPv4
goes but it _will_ work with IPv6.

As for the tailor made case where the customer buys a device actually made
for this, said device would also implement IPv6 for dual WAN. Plenty of
options for how the device could do that, including the possibility of
doing 1:1 stateless IPv6 NAT or simply presenting both prefixes to the LAN
and source route to the correct ISP.

Regards,

Baldur

Re: uPRF strict more

[Baldur Norddahl]

On Wed, 29 Sept 2021 at 22:07, Jean St-Laurent via NANOG 
wrote:

> Thanks a lot for sharing.
>
> So 100 Gbps at line rate with 80B frames is about ~150 Mpps.
>
> 100 Gbps at line rate with 208B frames is about ~60 Mpps.
>
> It's a significant penalty.
>

Full rate small packets would be an attack of some kind and could only
realistically arrive at your transit and peering ports. The customers
usually have slower (relatively) ports and a single customer could not
produce a rate of small packets that would be a concern. Therefore uRPF at
customer ports should not be a problem in this regard.

Regards,

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Sat, 25 Sept 2021 at 21:26, Owen DeLong via NANOG 
wrote:

> So the fact that:
>
> 2001:db8:0:1::5
> 2001:db8::1:0:0:0:5
>
> Are two different ways of representing the same address isn’t
> of any concern unless you’re making the mistake of trying to
> string wise compare them in their text-representation format.
> Both equate to the same uint128_t value.


If you adhere to RFC 5952 only the former is to be used (2001:db8:0:1::5).
Also strict RFC 5952 on any output will make a string compare ok because
there is only one way to print any address.

We should remember there are also multiple ways to print IPv4 addresses.
You can zero extend the addresses and on some ancient systems you could
also use the integer value.

You can even encounter IPv4 printed as IPv6 which is not too uncommon. Many
programs internally are IPv6 only and IPv4 is therefore mapped to IPv6. It
appears some people are forgetting this fact when proposing to drop IPv6.

Regards,

Baldur

Re: Rack rails on network equipment

[Baldur Norddahl]

The "niceness" of equipment does factor in but it might be invisible. For
example if you like junipers cli environment, you will look at their stuff
first even if you do not have it explicitly in your requirement list.

Better rack rails will make slightly more people prefer your gear, although
it might be hard to measure exactly how much. Which is probably the
problem.

Our problem with racking switches is how vendors deliver NO rack rails and
expect us to have them hanging on just the front posts. I have a lot of
switches on rack shelfs for that reason. Does not look very professional
but neither does rack posts bent out of shape.

My personal itch is how new equipment seems to have even worse boot time
than previous generations. I am currently installing juniper acx710 and
while they are nice, they also make me wait 15 minutes to boot. This is a
tremendous waste of time during installation. I can not leave the site
without verification and typically I also have some tasks to do after boot.

Besides if you have a crash or power interruption, the customers are not
happy to wait additionally 15 minutes to get online again.

Desktop computers used to be ages to boot until Microsoft declared that you
need to be ready in 30 seconds to be certified. And suddenly everything
could boot in 30 seconds or less. There is no good reason to waste techs
time by scanning the SCSI bus in a server that does not even have the
hardware.

Regards

Baldur



lør. 25. sep. 2021 21.49 skrev Andrey Khomyakov :

> Well, folks, the replies have certainly been interesting. I did get my
> answer, which seems to be "no one cares", which, in turn, explains why
> network equipment manufacturers give very little to no attention to this
> problem. A point of clarification is I'm talking about the problem in the
> context of operating a data center with cabinet racks, not a telecom closet
> with 2 post racks.
>
> Let me just say from the get go that no one is making toolless rails a
> priority to the point of shutting vendors out of the evaluation process. I
> am not quite sure why that assumption was made by at least a few folks.
> With that said, when all things being equal or fairly equal, which they
> rarely are, that's when the rails come in as a factor.
>
> We operate over 1000 switches in our data centers, and hardware failures
> that require a switch swap are common enough where the speed of swap starts
> to matter to some extent. We probably swap a switch or two a month.
> Furthermore, those switches several of you referenced, which run for 5+
> years are not the ones we use. I think you are thinking of the legacy days
> where you pay $20k plus for a top of rack switch from Cisco, and then sweat
> that switch until it dies of old age. I used to operate exactly like that
> in my earlier days. This does not work for us for a number of reasons, and
> so we don't go down that path.
>
> We use Force10 family Dell switches which are basically Broadcom TD2+/TD3
> based switches (ON4000 and ON5200 series) and we run Cumulus Linux on
> those, so swapping hardware without swapping the operating system for us is
> quite plausible and very much possible. We just haven't had the need to
> switch away from Dell until recently after Cumulus Networks (now Nvidia)
> had a falling out with Broadcom and effectively will cease support for
> Broadcom ASICs in the near future. We have loads of network config
> automation rolled out and very little of it is tied to anything Cumulus
> Linux specific, so there is a fair chance to switch over to Sonic with low
> to medium effort on our part, thus returning to the state where we can
> switch hardware vendors with fairly low effort. We are looking at Nvidia
> (former Mellanox) switches which hardly have any toolless rails, and we are
> also looking at all the other usual suspects in the "white box" world,
> which is why I asked how many of you care about the rail kit and I got my
> answer: "very little to not at all". In my opinion, if you never ask,
> you'll never get it, so I am asking my vendors for toolless rails, even if
> most of them will likely never get there, since I'm probably one of the
> very few who even brights that question up to them. I'd say network
> equipment has always been in a sad state of being compared to, well, just
> about any other equipment and for some reason we are all more or less
> content with it. May I suggest you all at least raise that question to your
> suppliers even if you know full well the answer is "no". At least it will
> start showing the vendors there is demand for this feature.
>
> On the subject of new builds. Over the course of my career I have hired
> contractors to rack/stack large build-outs and a good number of them treat
> your equipment the same way they treat their 2x4s. They torque all the
> screws to such a degree that when you have to undo that, you are sweating
> like a pig trying to undo one screw, eventually stripping it, so you have
> to drill t

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Sat, 25 Sept 2021 at 11:10,  wrote:

> Because IPv4 loopback is 127.0.0.1/8 and its usefull?
>

I am not sure why it is useful but nothing stops you from adding more
loopback addresses:

root@jump2:~# ip addr add ::2/128 dev lo
root@jump2:~# ping6 ::2
PING ::2(::2) 56 data bytes
64 bytes from ::2: icmp_seq=1 ttl=64 time=0.043 ms

While I am not sure what use extra addresses from the 127.0.0.0/8 prefix are
on the loopback, it is quite common for us to add extra global addresses
and then use that with proxy arp. Of course that is only necessary on IPv4
since IPv6 isn't so restrained that we have to save every last address bit
using tricks.


>
> - you can use nice short addreses like ::1234 for loopback
>

root@jump2:~# ip addr add ::1234/128 dev lo
root@jump2:~# ping6 ::1234
PING ::1234(::1234) 56 data bytes
64 bytes from ::1234: icmp_seq=1 ttl=64 time=0.046 ms

:-)


>   or ::1: for LL or ::1:0:1234 for RFC1918 like
>

With IPv6 you can use fe80::1: for link local and fd00::1:0:1234 for
your RFC1918 like setup. And then you can use 1:1 NAT to transform that to
GUA on the router. Even NAT, if you insist on using it, is better with IPv6.

The confusion here appears to be that auto generated link local prefixes
are long with many hex digits. But compared to the new proposal, which
could have no auto generated link local due to having too few bits, there
is nothing that stops you from manually assigning link local addresses. It
is just that nobody wants to bother with that and you wouldn't either.

Example:

root@jump2:~# ip addr add fe80::1:/64 dev eth0
root@jump2:~# ping6 fe80::1:%eth0
PING fe80::1:%eth0(fe80::1:) 56 data bytes
64 bytes from fe80::1:: icmp_seq=1 ttl=64 time=0.033 ms



> ND is new thing and it requires new things to protect it from attacks?
>

I am not aware of any NDP attacks that would be any different if based on
ARP. Those two protocols are practically the same.

Regards,

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Thu, 23 Sept 2021 at 21:48, Christopher Morrow 
wrote:

> This sounds like very naive nat state management behavior.
> Ideally, you'd  be able to maintain state of:
> original-src/dst/ports/proto -> in-interface/external ip/port/proto
>

What you describe is called symmetric NAT and is the kind which has the
highest impact on customers. Many NAT traversal techniques fail to work
with this type of NAT. STUN does not work with symmetric NAT.

You purposely want a lesser NAT type which makes NAT traversal easy. This
enables more applications to function despite the NAT. Almost every CPE
avoids symmetric NAT for one of the lesser types for this reason.


>
> unless some internal/original src is double using port/proto ... you
> should really
> have the ability to nat quite a large number of things to a single ipv4
> address.
>

There is no point in optimizing your customer per public CGN IP address
ratio and multiple reasons for not doing so. At some point you will
experience attacks on the CGN address or blacklisting in online games and
so on. Having less customers affected each time that happens is better.

In our network approximately 10% of new customers signed up within the last
year have asked to escape the CGN and get a public IP. This means I need
100 IPv4 addresses per 1000 new customers. Plus 4 IPv4 addresses for the
CGN server. What difference would it make if we could optimize those 4
addresses to be just 2 or maybe 1? What if it caused just a couple more
customers to request a public IP address because they got annoyed by a more
restrictive NAT or by failed sessions?

Regards,

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

tor. 23. sep. 2021 01.39 skrev Colton Conor :

> Where does this "You can only have about 200-300 subscribers per IPv4
> address on a CGN." limit come from? I have seen several apartment
> complexes run on a single static IPv4 address using a Mikrotik with
> NAT.
>

It is our observation as the limit before you regularly run out of ports
using Linux as a CGN server.

It will still work if you have more users on an IP. The users will just
experience session failures at peak. Low levels of that might show up as
pictures that fail to load on a web page and be ok when the user reloads.
This will increase the number of support calls and the number of customers
that asks to escape the CGN. Or people will live with it and just think
that the Internet connection is low quality.

Regards

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Wed, 22 Sept 2021 at 16:48, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Today, as /24 can afford hundreds of thousands of subscribers
> by NAT, only very large retail ISPs need more than one
> announcement for IPv4.
>

You can only have about 200-300 subscribers per IPv4 address on a CGN. If
you try to go further than that, for example by using symmetric NAT, you
will increase the number of customers that want to get a public IPv4 of
their own. That will actually decrease the combined efficiency and cause
you to need more, not less, IPv4 addresses.

Without checking our numbers, I believe we have at least 10% of the
customers that are paying for a public IPv4 to escape our CGN. This means a
/24 will only be enough for about 2500 customers maximum. The "nat
escapers" drown out the efficiency of the NAT pool.

The optimization you need to do is to make the CGN as customer friendly as
possible instead of trying to squeeze the maximum customers per CGN IPv4
address.

Perhaps IPv6 can lower the number of people that need to escape IPv4 nat.
If it helps just a little bit, that alone will make implementing IPv6 worth
it for smaller emerging operators. Buying IPv4 has become very expensive.
Yes you can profit from selling a public IPv4 address to the customer, but
there is also the risk that the customer just goes to the incumbent, which
has old large pools of IPv4 and provides it for free.

Regards,

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

ons. 15. sep. 2021 19.37 skrev Owen DeLong via NANOG :

>
>
> > On Sep 15, 2021, at 09:31 , Masataka Ohta <
> mo...@necom830.hpcl.titech.ac.jp> wrote:
> >
> > Baldur Norddahl wrote:
> >
> >>>> But in fact with local number portability, you cannot rely on the
> county
> > 
> >>>> code to tell you where to route a telephone call anymore.
> >>>
> >>> Not. With geographical aggregation, you may route a call
> >>> *anywhere* in the destination country.
> >
> >> You mean anywhere in the world. Calls to my number reach my cell phone
> no
> >> matter where I go.
> >
> > You are confusing number portability and call forwarding.
> >
> >   Masataka Ohta
>
> Not exactly… He is confusing number portability and roaming.
>

I am not confusing anything. Nobody said anything about number portability
in the above quote.

Just pointing out that some assumptions about how voice call works is not
true. Nor would it be smart to send traffic to another part of the world
unnecessarily. Local calls stay local even when roaming. Please remember
that signaling and voice is separate in the telco world which is why it
compares poorly with IP. Except for the LISP protocol which does something
similar.

Regards

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Wed, 15 Sept 2021 at 06:38, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Shane Ronan wrote:
>
> > But in fact with local number portability, you cannot rely on the county
> > code to tell you where to route a telephone call anymore.
>
> Not. With geographical aggregation, you may route a call
> *anywhere* in the destination country.
>

You mean anywhere in the world. Calls to my number reach my cell phone no
matter where I go.


>
> Number portability database is looked up after the call
> reaches the destination country, which will be used for
> further intra-national routing, which do not affect
> country-wise aggregation of international routing table.
>
>
Actually the GSM system will query the HLR to find out where to really
route the call. Much like LISP actually.

Regards,

Baldur

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Mon, Sep 13, 2021 at 8:22 PM Randy Bush  wrote:

> real compatibility with ipv4 was disdained.  the transition plan was
> dual stack and v4 would go away in a handful of years.  the 93
> transition mechanisms were desperate add-ons when v4 did not go away.
> and dual stack does not scale, as it requires v4 space proportional to
> deployed v6 space.
>

What I find most peculiar about this whole rant (not just yours but the
whole thread) is that I may be the only one who found implementing IPv6
with dual stack completely trivial and a non issue? There is no scale issue
nor any of the other rubbish.

Some say what we have is not a true dual stack setup because we run MPLS
and the IPv6 mostly lives inside L2VPN or L3VPN tunnels. Most of our
network gear has no idea what an IPv6 address is. But neither does that
equipment touch the public IPv4 internet. Nevertheless we configure our
IPv4 and IPv6 both only on our few edge Juniper MX devices and that is it.
I do not believe IPv6 has 100 config lines in my network total.

For all I care we already have a perfect working system with IPv4+CGN+IPv6.
The CGN part was the most troublesome, not the IPv6.

Regards,

Baldur

Re: Never push the Big Red Button (New York City subway failure)

[Baldur Norddahl]

A nearby datacenter once lost power delayed because someone hit the switch
to transfer from city power to generator power and then failed to notice.
The power went out the day after when there was no fuel left.

On Fri, Sep 10, 2021 at 9:24 PM Matthew Huff  wrote:

> Since we are telling power horror stories…
>
>
>
>
>
> How about the call from the night operator that arrived at 10:00pm asking
> “Is there any reason there is no power in the data center?”
>
>
>
> Turns out someone had plugged in a new high end workgroup laser printer to
> the outside wall of the datacenter. The power receptacle was wired into the
> data center’s UPS and completely smoked the UPS. Luckily the static
> transfer switched worked, but the three mainframes weren’t’ happy…
>
>
>
>
>
> Or
>
>
>
> Our building had a major ground fault issue that took years to find and
> resolve. We got hit with lightning that caused the mainframe to fault and
> recycle…and two minutes in, we got hit by lightning again. When the system
> failed to start, we called IBM support. When we explained what happened
> there was a very long pause…then some mumbling off phone, then the manager
> got on the line and said someone would be flying out and be onsite within
> 12 hours. We were down for 3 days, and got fined $250,000 by the insurance
> regulators since we couldn’t pay claims.
>
>
>
> *Matthew Huff* | Director of Technical Operations | OTA Management LLC
>
>
>
> *Office: 914-460-4039*
>
> *mh...@ox.com  | **www.ox.com *
>
>
> *...*
>
>
>
> *From:* Chris Kane 
> *Sent:* Friday, September 10, 2021 3:16 PM
> *To:* Christopher Morrow 
> *Cc:* Matthew Huff ; nanog@nanog.org
> *Subject:* Re: Never push the Big Red Button (New York City subway
> failure)
>
>
>
> True EPO story; maintenance crew carrying new drywall into the data center
> backed into the EPO that didn't have a cover on it. One of the most
> eerie sounds in networking...a completely silent data center.
>
>
>
> -chris
>
>
>
> On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow <
> morrowc.li...@gmail.com> wrote:
>
>
>
>
>
> On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff  wrote:
>
> Reminds me of something that happened about 25 years ago when an
> elementary school visited our data center of the insurance company where I
> worked. One of our operators strategically positioned himself between the
> kids and the mainframe, leaned back and hit it's EPO button.
>
>
>
> Or when your building engineering team cuts themselves a new key for the
> 'main breaker' for the facility... and tests it at 2pm on a tuesday.
>
> Or when that same team cuts a second key (gotta have 2 keys!) and tests
> that key on the same 'main breaker' ... at 2pm on the following tuesday.
>
>
>
> 
>
>
>
> not fakenews, a real story from a large building full of gov't employees
> and computers and all manner of 'critical infrastructure' for the agency
> occupying said building.
>
>
>
> Matthew Huff | Director of Technical Operations | OTA Management LLC
>
> Office: 914-460-4039
> mh...@ox.com | www.ox.com
>
> ...
>
> -Original Message-
> From: NANOG  On Behalf Of Sean
> Donelan
> Sent: Friday, September 10, 2021 12:38 PM
> To: nanog@nanog.org
> Subject: Never push the Big Red Button (New York City subway failure)
>
> NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
> OUTAGE ISSUE ON AUGUST 29, 2021
> Key Findings
> September 8, 2021
>
>
>
> https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf
>
> Key Findings
> [...]
>
> 3. Based on the electrical equipment log readings and the manufacturer’s
> official assessment, it was determined that the most likely cause of RCC
> shutdown was the “Emergency Power Off” button being manually activated.
>
> Secondary Findings
>
> 1. The “Emergency Power Off” button did not have a protective cover at the
> time of the shutdown or the following WSP investigation.
>
> [...]
> Mitigation Steps
>
> 1. Set up the electrical equipment Control and Communication systems
> properly to stay active so that personnel can monitor RCC electrical
> system operations.
>
> [...]
>
>
>
>
> --
>
> Chris Kane
>

Re: IPv6 woes - RFC

[Baldur Norddahl]

On Mon, Sep 6, 2021 at 2:55 PM Bjørn Mork  wrote:

> And cellular providers give you a single /64.  Not even useful as IPv6
> access for anything larger than a single handset.  Extending that /64 to
> something you can use is non-trivial.  How many providers have actually
> done that?
>

I am sitting here on a 4G broadband connection provided by the
cellular provider "3". I am using the Huawei B535-232 CPE provided to me by
"3" as part of the contract. My IPv6 configuration, which I
received completely automatic without having to do anything or even know
about it:

Router IPv6: 2a02:aa7:4620:e76b:5a02:3ff:fe04:506
My computer connected through a LAN cable to the
router: 2a02:aa7:4620:e76b:1870:3bef:4272:5
My android cell phone connected via wifi:
2a02:aa7:4620:e76b:d199:f9b2:7103:3a05

As should be obvious the /64 provided extends to something larger than a
single handset here in a completely trivial way. The CPE is simply bridging
the /64 provided.

Yes I would be more happy with a prefix delegation of some size but this
works too. It is still better than the IPv4 which has to go through NAT.

Regards,

Baldur

Re: An update on the AfriNIC situation

[Baldur Norddahl]

On Sat, Aug 28, 2021 at 1:07 AM Bill Woodcock  wrote:

> > On Aug 28, 2021, at 12:48 AM, Baldur Norddahl 
> wrote:
> > just to point out it is not just one guy but a whole region doing
> business like that.
>
> You’re saying a whole region consists of parties who don’t route IP
> traffic?
>
>
None of the regions require that the IP allocations are used for routing IP
traffic. Aside from that, I am saying that RIPE no longer has a "need"
policy which means you can have as much IP space as you want without
providing any need to own this space. In addition we have IP brokers that
expressly buys IP blocks for the purpose of selling - not routing - the
space.



> If not, you’re making a false equivalency.
>
> > In the RIPE region we had a run with many parties that created fake LIRs
> to get an extra /22 assignment.
>
> That’s unfortunate, and I hope RIPE revokes any allocations which were
> made under false pretenses and are being currently used in ways that
> violate RIPE’s current policies.  As AfriNIC does.
>

None of the regions will do anything like that as long as current policies
are fulfilled. As RIPE no longer has a policy of showing any need, no such
revocations will happen. This is a case only because the guy is using space
from AfriNIC.


>
> > Did they steal that any less than this guy?
>
> I believe the blocks in question are:
>
> 154.80.0.0/12
> 45.192.0.0/12
> 156.224.0.0/11
> 154.192.0.0/11
>
> So, yes, 6,144 times less.
>

Nah, what happened in RIPE with the /22 blocks was on a similar or larger
scale. Remember we had a whole /8 plus some for that policy.


>
> > This guy is a small fish compared to the robbery done by so many others.
>
> If you know of someone who’s fraudulently acquired _more_ than 6.3M IPv4
> addresses, and is profiting from their being used in contravention of RIR
> policy, I very much encourage you to request that your RIR perform a
> compliance audit.
>
>
Say we have an ISP with an allocation of 8 million addresses in a country
with 5 million people, including babies, a case can easily be made that
they have approximately that many IPv4 addresses they do not really need.
Maybe they will sell them for a profit some day. But RIPE _will not revoce_
those addresses. Does not matter that the business plans or whatever they
did to get that many were totally fake back then. And this happened
everywhere. Do not be naive, it was also going on in every other region.

In fact, a case can be made for the whole ARIN region of collectively
hoarding IPv4 space which now can be sold for a profit.

Or what about going back to 2011 when Microsoft bought a large IPv4 block
from someone who was not routing said block? This guy did not invent the
business model!

Fact is IPv4 was a lottery and the winnings have been distributed. A lot of
those that played did cheat. They are not going to pay for that now.

What perhaps makes this case different is that he is robbing AfriNIC. I am
afraid they are set up to be robbed as the last RIR to still have IPv4
available. Perhaps AfriNIC should just auction it away and use the money to
help the internet in Africa in other ways.

Regards,

Baldur

Re: An update on the AfriNIC situation

[Baldur Norddahl]

On Sat, Aug 28, 2021 at 12:13 AM Bill Woodcock  wrote:

> Well, I hope not _many_ other parties.  I guess we’re not talking about “a
> completely different case” after all, then?  Bear in mind that this guy is
> in _no way_ part of the Internet ecosystem.  He is _solely_ extracting rent
> by renting something he stole from us, back to us.  If you’re saying,
> “Well, is that really so bad? This guy steals my car, but at least he’s
> willing to rent it back to me… doesn’t that happen all the time?”  No, not
> so much.
>

That is completely legal and within the rules in the RIPE region now. Yes I
know not AfriNIC but just to point out it is not just one guy but a whole
region doing business like that. We even have an official registry of
companies that buy and sell IP address space.

And by many parties I mean _most_ parties that were part of the system just
before we ran out of IP addresses. Everyone knew it was the last chance so
you had to be an idiot not to get as much address space as possible. Here
in my country the largest ISP got an assignment so big, that they can use 2
IPs for every person living in the country. They do not have a market share
that large, but I suppose they could make up a business plan to lay out
plans for that. And now they can either offer IPv4 services that smaller
and younger ISPs can not or they can sell/rent the space and profit in
exactly the same way as our hypothetical friend here.

In the RIPE region we had a run with many parties that created fake LIRs to
get an extra /22 assignment. Did they steal that any less than this guy?

I believe all of that stinks. It is just that this guy is a small fish
compared to the robbery done by so many others.

Regards,

Baldur

Re: An update on the AfriNIC situation

[Baldur Norddahl]

Hello

I know nothing about this case although it sounds like this guy needs to be
stopped. However, let's pretend that I am talking about a completely
different case.

A guy sometime in the past acquired some large blocks of IP addresses. He
was not completely honest at the time so he got more than he should have.
At the time this meant that not all addresses were in use.

Now in 2021 he is profiting from selling or leasing out these addresses.
This is clearly unfair as he lied to get them back then. However this means
the addresses are actually in use _now_.

BUT - how is this so different from what many many other parties have done?
I think we all know some huge ISPs that got much larger blocks than
strictly needed, and which now are profiting directly or indirectly. I am
not even talking about those that got huge blocks back when everyone
thought IPv4 was unlimited. But ISPs that lied about current needs and
exacareted business plans and so on, to get as large blocks as
possible from RIRs.

Yes I understand that the case is also about using blocks in a different
region, but that too is something many others have done.

The guy might be a scoundrel but is he in good company?

Regards,

Baldur

Re: "Tactical" /24 announcements

[Baldur Norddahl]

On Fri, Aug 13, 2021 at 10:53 PM Amir Herzberg  wrote:

>
> I think it isn't the same.
>

I am still not sure but maybe I misunderstood what you originally said. It
is probably not important.


> I think that the NANOG (or in general, operators) community may do well to
> state the `/24 rule' clearly in a BCP, preferably an RFC. A mismatch in the
> most-specific rule can definitely allow different problems (and attacks).
> As mentioned above, RIPE has essentially done this (although could be more
> explicit). I've seen a similar /48 rule for IPv6, btw.
>

I am not sure how big a problem this is. We only had this one case that I
described and it was easily fixed by allowing that one prefix from our
transit. The peer also offered to fix their announcement. But we did not
run with it for very long because we only reduced our routing table to
debug a different problem.

Maybe we could have a community or other mechanism to mark the few routes
that can not be dropped in exchange for a default route.

For all the stub networks out there we should be able to aggressively
filter routes without much harm.

Regards,

Baldur

Re: "Tactical" /24 announcements

[Baldur Norddahl]

On Fri, Aug 13, 2021 at 3:54 AM Amir Herzberg  wrote:

> On Thu, Aug 12, 2021 at 4:32 PM Baldur Norddahl 
> wrote:
>
>>
>>
>> On Thu, Aug 12, 2021 at 7:39 PM Amir Herzberg 
>> wrote:
>>
>>> Bill, I beg to respectfully differ, knowing that I'm just a researcher
>>> and working `for real' like you guys, so pls take no offence.
>>>
>>> I don't think A would be right to filter these packets to 10.0.1.0/24;
>>> A has announced 10.0.0.0/16 so should route to that (entire) prefix, or
>>> A is misleading its peers.
>>>
>>
>> You are right that it is wrong but it happens. Some years back I tried a
>> setup where we wanted to reduce the size of the routing table. We dropped
>> everything but routes received from peers and added a default to one of our
>> IP transit providers. This should have been ok because either we had a
>> route to a peer or the packet would go to someone who had the full routing
>> table, yes?
>>
>
> Baldur, thanks, but, sorry, this isn't the same, or I miss something.
>

I think it is exactly the same? Our peer is advertising a prefix for which
they will not route all addresses covered. Is that route not then a lie?
Should they not have exploded the prefix so they could avoid covering the
part of the prefix they will not accept traffic to? (ps: not arguing they
should!)


> If I get you right, you dropped all announcements from _providers_ except
> making one provider your default gateway (essentially, 0.0.0.0/0). But
> this is very different from what I understood from what Bill wrote. Your
> change could (and, from what you say next, did) cause a problem if one of
> these announcements you dropped from providers was a legit subprefix of a
> prefix announced by one of your peers, causing you to route to the peer
> traffic whose destination is in the subprefix.
>

Your understanding is correct. But this is just the way we ended up in that
situation. Does not change the fact that we got a route from a peer that we
believed we could use, but turns out part of that announcement was a lie.

Consider that everyone filters received routes. The most common is to
filter at the /24 level but nowhere is there a RFC stating that /24 is
anything special. So what if I was to filter at a different level, say /20
? The same thing would happen, we would drop the "/24 exception route" and
use the route that is a lie.

Regards,

Baldur

Re: "Tactical" /24 announcements

[Baldur Norddahl]

On Thu, Aug 12, 2021 at 7:39 PM Amir Herzberg  wrote:

> Bill, I beg to respectfully differ, knowing that I'm just a researcher and
> working `for real' like you guys, so pls take no offence.
>
> I don't think A would be right to filter these packets to 10.0.1.0/24; A
> has announced 10.0.0.0/16 so should route to that (entire) prefix, or A
> is misleading its peers.
>

You are right that it is wrong but it happens. Some years back I tried a
setup where we wanted to reduce the size of the routing table. We dropped
everything but routes received from peers and added a default to one of our
IP transit providers. This should have been ok because either we had a
route to a peer or the packet would go to someone who had the full routing
table, yes?

So we got complaints. One was a company who would advertise a /20 on a
peering with us. But somewhere else far away they had a site from where
they would announce a /24 from the same prefix. With no internal routing
between the peering site with the /20 to the other site with the /24. We
therefore lost the ability to communicate with that /24.

You see variants of this. For example a large telco has a /16 from which
they many years ago allocated a /24 to a multihomed customer. This customer
left but took their /24 with them. This fact will seldom make the large
telco split up their /16. They will keep it as a /16 but will no longer
route to that /24. The question is also if we really would want a large
telco to explode a large subnet due to this case.

Regards,

Baldur

Re: "Tactical" /24 announcements

[Baldur Norddahl]

man. 9. aug. 2021 22.13 skrev Grzegorz Janoszka :

> On 2021-08-09 17:47, Billy Croan wrote:
> > How does the community feel about using /24 originations in BGP as a
> > tactical advantage against potential bgp hijackers?
>
> RPKI is more effective than a competing /24. Unless they hijack you ASn
> as well.
>

You will usually get an as path length advantage even if they do hijack
your asn.

Regards

Baldur

Re: russian prefixes

[Baldur Norddahl]

On Wed, Jul 28, 2021 at 11:29 PM Randy Bush  wrote:

>
> https://www.businessinsider.com/russia-cuts-self-off-from-global-internet-tests-defenses-rbc-2021-7
> says "Russia disconnected itself from the rest of the internet, a test
> of its new defense from cyber warfare, report says"
>

Would that even be effective? Surely a state sponsored cyber warfare attack
can use any domestic internet connection within Russia to continue the
attacks.

Re: Anycast but for egress

[Baldur Norddahl]

Here is what I think would happen if you were to try this setup. Let's
assume you deployed in eu-west-2 (London) and eu-central-1 (Frankfurt). You
would find that you could successfully connect to a number of networks but
also that some of them would work from the "wrong" site. Eg. you would have
clients in London that require you to use the Frankfurt instance to connect
to. You would also find a lot of networks that you could not connect to
from either site. And you would have some instability where some networks
at random switch between these states. For example you could have a client
that one day works from London, the next day it is Frankfurt and then
someday it won't be working at all.

As you add more sites, you also add more ways for the traffic to end up in
the wrong places. You could have something that works with just the two
sites above but then when you add eu-west-1 (Ireland) you could suddenly
not connect to them from any of the three sites.

There IS a possible solution which is to tunnel the traffic back to the
correct site. This still requires you to use unique IP addresses for each
site, but all addresses could be in the same subnet. You would have IP
a.b.c.1 to be London, a.b.c.2 Frankfurt, a.b.c.3 Ireland. Then if London
receives traffic to a.b.c.2 you would have a tunnel to send the traffic to
Frankfurt.

Regards,

Baldur


On Wed, Jul 28, 2021 at 11:07 AM Baldur Norddahl 
wrote:

>
>
>> > On Jul 27, 2021, at 17:20, Vimal  wrote:
>> > Yes, this makes sense as the destination can be anywhere around the
>> world, and that routing is asymmetric as others mentioned.  However, if the
>> destination service is "close" (in the routing metric sense) to the
>> initiating host, anycast return IP ought to work well, right?  I understand
>> this is a very important caveat and impractical to implement correctly in
>> the real world.
>>
>
> No, there is no such thing as "close". You could have a direct peering
> with some ISP and have them still deliver the responses on the other side
> of earth. You do not control the routing of other networks and can not be
> sure what they will do.
>
> For larger networks you may also have multiple peering points. Say you
> have a peering with them in city A and city B. How do you know which of
> their IP ranges are closer to A or B? You don't. And the same goes for
> them, they have no idea if you prefer A or B. Therefore you could select A
> and they may reply to B. They may even load balance between A and B if you
> are really unlucky.
>
> Routing is asymmetric. That means you have absolutely no idea where the
> replies end up going. Often it will not be what you think is "close".
>
> I do not run anycast, but I understand that the usual way of dealing with
> these issues is to do as little as possible with anycast before redirecting
> to a unicast address. For example you could have just your DNS on anycast
> and each site would reply with unique unicast addresses. Since DNS is just
> a single pair of UDP request/response, with the first packet originating
> from a unicast client, this works well.
>
> Regards,
>
> Baldur
>
>
>

Re: Anycast but for egress

[Baldur Norddahl]

>
> > On Jul 27, 2021, at 17:20, Vimal  wrote:
> > Yes, this makes sense as the destination can be anywhere around the
> world, and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
>

No, there is no such thing as "close". You could have a direct peering with
some ISP and have them still deliver the responses on the other side of
earth. You do not control the routing of other networks and can not be sure
what they will do.

For larger networks you may also have multiple peering points. Say you have
a peering with them in city A and city B. How do you know which of their IP
ranges are closer to A or B? You don't. And the same goes for them, they
have no idea if you prefer A or B. Therefore you could select A and they
may reply to B. They may even load balance between A and B if you are
really unlucky.

Routing is asymmetric. That means you have absolutely no idea where the
replies end up going. Often it will not be what you think is "close".

I do not run anycast, but I understand that the usual way of dealing with
these issues is to do as little as possible with anycast before redirecting
to a unicast address. For example you could have just your DNS on anycast
and each site would reply with unique unicast addresses. Since DNS is just
a single pair of UDP request/response, with the first packet originating
from a unicast client, this works well.

Regards,

Baldur

Re: 100G, input errors and/or transceiver issues

[Baldur Norddahl]

You could also enable FEC on the link. This will remove any errors until
the link quality is really far gone.

Regards

Baldur


man. 19. jul. 2021 20.06 skrev Graham Johnston :

> Thank you all for the consensus. What I hear from you is that 100G takes
> more care to operate error free, as compared to 10G, which wasn't
> surprising to me. Also, that generally, we should be able to operate
> without errors, or certainly less than I'm currently observing, and that
> connector and transceiver interface cleanliness is our first likely point
> of investigation.
>
> Thanks to all who responded.
>
> On Mon, 19 Jul 2021 at 12:58, Jared Mauch  wrote:
>
>>
>>
>> > On Jul 19, 2021, at 1:50 PM, Saku Ytti  wrote:
>> >
>> > On Mon, 19 Jul 2021 at 20:19, Graham Johnston
>> >  wrote:
>> >
>> >> I don't at this point have long term data collection compiled for the
>> issues that we've faced. That said, we have two 100G transport links that
>> have a regular background level of input errors at ranges that hover
>> between 0.00055 to 0.00383 PPS on one link, and none to 0.00135 PPS (that
>> jumped to 0.03943 PPS over the weekend). The range is often directionally
>> associated rather than variable
>> >
>> > On typical 100G link we do not get single FCS error in a typical day.
>> > However Ethernet spec still allows very high error rate of 10**-12. So
>> > 1 error per 1Tb (b not B). I.e. 1 error per 10s, or 0.1PPS would be
>> > in-spec. We see much better performance to this and vendors generally
>> > accept lower error rates as legitimate errors.
>> >
>>
>> I will confirm my experience with this at $dayjob as well.  We see
>> interfaces with no errors over much longer periods of time inclusive of
>> several of the technology options.  If you are seeing errors, there’s
>> likely something wrong like unclean fiber or bad optic/linecard etc.
>>
>> - Jared
>>
>>

Re: Do you care about "gray" failures? Can we (network academics) help? A 10-min survey

[Baldur Norddahl]

We had a line card that would drop any IPv6 packet with bit #65 in the
destination address set to 1. Turns out that only a few hosts have this bit
set to 1 in the address, so nobody noticed until some debian mirrors
started to become unreachable. Also webbrowser are very good at switching
to IPv4 in case of IPv6 timeouts, so nobody would notice web hosts with the
problem. And then we had to live with the problem for a while because the
device was out of warranty and marked to be replaced, but you do not just
replace a router in a hurry unless you absolutely need to.

You do not expect this kind of issue and a lot of time was spent trying to
find an alternate explanation for the problem.

Regards,

Baldur

Re: FreeBSD's ping Integrates IPv6

[Baldur Norddahl]

søn. 4. jul. 2021 12.45 skrev Mark Tinka :

>
>
> On 7/4/21 05:51, Owen DeLong wrote:
>
> > Linux did this quite some time ago. I guess BSD is just now catching up.
>
> Been nearly 14 years since I last operated a Linux machine.
>

Some Juniper gear is Linux hypervisor :-)

Re: Layer 2 based anycast - Kind like GLBP - Research

[Baldur Norddahl]

tor. 1. jul. 2021 21.06 skrev William Herrin :

>
>
> From what I understand of EVPN, it's about creating something
> equivalent to VLANs across a distributed virtual server
> infrastructure. Basically like what Amazon does under the hood for its
> virtual private cloud. Since you're trying to get the machines to
> appear on the same subnet, not separate them to different subnets, I
> don't think it's what you're looking for.
>


EVPN creates a virtual layer 2 domain, aka a vlan, that can span the
internet or be used on a plain old layer 2 switch. It uses vxlan or mpls
tunnels and BGP to coordinate. EVPN has support for multiple active/active
exits, something almost like lacp. There can be load balancing using layer
3 headers as key, which might be exactly what OP is looking for.

EVPN elimates layer 2 flooding by keeping a database of MAC addresses in
BGP. Otherwise it behaves exactly like a vlan with extra features.

>

Re: Layer 2 based anycast - Kind like GLBP - Research

[Baldur Norddahl]

On Thu, Jul 1, 2021 at 8:04 PM Douglas Fischer 
wrote:

> I friend Suggested that EVPN could help-me, but I must confess that is a
> hard topic to me.
> Unless it can be used depending exclusively on software (no special
> hardware required), it won't fit.
>
>
Linux has EVPN support. Or you could use network switches with EVPN support.

EVPN will let each server or direct connected switch be a hidden layer 3
gateway. The application will not know about it and it will appear to be
layer 2 but routable like layer 3. Which means you can do anycast at layer
3. Physically the packets are moved in tunnels so you can use any layer 2
or layer 3 hardware at your choice.

Regards,

Baldur

Re: Can somebody explain these ransomwear attacks?

[Baldur Norddahl]

fre. 25. jun. 2021 21.33 skrev Aaron C. de Bruyn via NANOG :

> On Fri, Jun 25, 2021 at 10:43 AM Tom Beecher  wrote:
>
>> Incompetent insurance companies combined with incompetent IT staff and
>>> under-funded IT departments are the nexus of the problem.
>>>
>>
>> Nah, it's even simpler. It's just dollars all around. Always is.
>>
>
> Agreed.
>
>
>> From this company's point of view, the cost to RECOVER from the problems
>> is so much smaller than it would be to prevent the problems from happening
>> to begin with, so they are happy to let you guys handle it. From the
>> insurance company's point of view, they are collecting premiums, but no
>> claims are being filed, so they have no incentive to do anything
>> differently.
>>
>
> I'm sure that'll change drastically if either of these conditions are true:
> * A claim is filed
> * An audit is required
> * Ransomware surges throughout 2021 and payouts go through the roof
>
> I think it's reasonable to expect at least one of those things will happen
> in the next year.
>
> -A
>

Or they do business in the EU where huge fines are becoming the norm. The
ransomware does not matter but the implied data breach does.

Re: IPv6 and multicast listener discovery

[Baldur Norddahl]

If you enable MLD snooping on your switches, the switch will block
multicast going out irrelevant ports. The idea is to reduce broadcast in a
broadcast domain.

On Fri, Jun 4, 2021 at 11:03 PM William Herrin  wrote:

> Howdy,
>
> Question for those more versed in IPv6 than I: Is there any harm from
> dropping ICMPv6 multicast listener discovery reports in a network
> which does NOT use any multicast routing (i.e. only uses multicast
> which stays within the local link). I see a LOT of idle node chatter
> in the form of these reports which, of course, flood every station
> since they are themselves multicast. As far as I can tell they are
> used only to tell a multicast router whether to repeat a particular
> set of multicast packets to the instant link. Which in my network is
> -never- because there are no routed multicast packets to be repeated.
>
> Regards,
> Bill Herrin
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On Fri, Jun 4, 2021 at 1:49 PM Mike Hammett  wrote:

> Assuming you were able to get the maximum capacity (you don't for a
> variety of reasons), the maximum capacity of a given access point is 1.2
> gigabit/s. On a 2:1 ratio, that's about 800 megs down and 400 megs up.
>
>
Here is a graph of traffic from approx 200 GPON customers, with a mix of
200/200 and 1000/1000 subscription types:

https://oz9h.dk/graph.png

Something tells me that would also work just fine with wireless operating
at link speed of 1,2 Gbps. You would of course not be able to do 1000 Mbps
upload with a link of 400 up, but you would be able to sell 200/200 no
problem. The limit would be downstream capacity, not upstream.

Regards,

Baldur

Re: Muni broadband sucks (was: New minimum speed for US broadband connections)

[Baldur Norddahl]

On Fri, Jun 4, 2021 at 2:53 AM Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Baldur Norddahl wrote:
>
> > Sorry but that claim is completely wrong. Cabling cost scales linearly
> with
> > the number of cores.
>

My apology to Masataka Ohta for my too strong wording by calling you wrong.
The moderators put me in place. I wanted to say I disagree with the claim.


> Most of cabling cost is cost to lay cables. Backhoe costs.
> Space factor of a fiber cable is negligible if you put a
> cable into utility tunnels which is wide, especially when
> tunnels were used for copper cables of POTS.
>

It is true that trenching costs are higher than the cables themselves. But
that does not mean the cables are cheap and that it is an
insignificant cost. Cables + duct is about 20% of our cost to lay down the
network. Not having huts with active equipment spread all around is also a
huge cost saving that can not be ignored.

 > The cost of 144 is not
>  > double that of 72.  288 is not double the cost of 144.
>
> Yup. When PON was first conceived several tens of years ago, core
> cost a lot. But, today...
>

I should point out that I probably buy more cable than most. The exact
pricing is of course not public, but lets say a core cost somewhere between
1 to 2 USD cents per meter. Then you simply multiply up to get an
approximate price of the cable. Holds true for cables with more than about
12 cores. This is because with larger cables the cost of the cores dominate
the price of the cable. When you buy as much as we do, you do not really
get a huge rebate for buying more cores in a single cable - we already buy
insane amounts of core - it is just distributed in more cables.

The moderator is right in that we do not seem to progress much here in this
discussion. So lets agree to disagree. But let me get this closing comment
in anyway... the guy that actually builds PON networks says he does so,
because it is significantly cheaper. We have no other motivations as our
network is not open to third parties in any case. Our motivation is to stay
profitable.

Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On Thu, Jun 3, 2021 at 11:46 PM Mike Hammett  wrote:

> 2.4 gigabit per channel, but only 1.2 gigabit from a given access point.
>
> Most often, WISPs choose down\up ratios between 85/15 and 66/34 and then
> sell plans appropriately. If we're now required to have a symmetric 100
> megs, you'll be robbing even more of the downstream for the upstream. Why
> would you do that? So that you're relatively capable of providing what
> you're selling. The alternative is gross oversubscription.
>

66/34 is 2:1 or exactly the same as GPON (2.4 down, 1.2 up). We sell 1000
symmetrical on that GPON and the customers are happy. You would have much
less oversubscription with 100/100 on a 1.2 Gbps wireless with 66:34
down/up ratio, than we are doing with GPON and 1000/1000. We are also doing
128 customers on a single OLT port.

Remember that a single customer only adds a few Mbps peak to your bandwidth
usage.

Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On Thu, Jun 3, 2021 at 2:40 PM Forrest Christian (List Account) <
li...@packetflux.com> wrote:

> I think you're really out of touch with what is going on in the WISP space.
>
> See the following product as an example:
>
>
> https://www.cambiumnetworks.com/products/pmp-450/5-ghz-pmp-450m-fixed-wireless-access-point/
> 14x14 beam-steering Massive Multi-User MIMO.   This is able to talk, in
> the same channel, at the same time, to up to 7 endpoints using both
> vertical and horizontal polarities at the same time.  Total throughput
> per 40Mhz channel: 1.2Gb/s per AP.
>
> Because of the TDMA synchronization, you can actually hang two of these on
> the same tower front to back using the same channel.   So 2.4Gb/s per
> Frequency.  And there are dozens of channels available at this point.
>
>
But isn't that just proving my point? If you can do 2,4 Gbps per frequency,
why are the WISPs whining about a 100 Mbps requirement?!

Regards,

Baldur

Re: Muni broadband sucks (was: New minimum speed for US broadband connections)

[Baldur Norddahl]

On Thu, Jun 3, 2021 at 5:41 PM Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> As cabling cost is mostly independent of the number of cores in a
> cable, as long as enough number of cores for single star are provided,
> which means core cost is mostly cabling cost divided by number of
> subscribers, single star does not cost so much.
>

Sorry but that claim is completely wrong. Cabling cost scales linearly with
the number of cores. A 192 core cable is approximately twice the price of a
96 core cable. Only at very low core count does this break up somewhat. A
12 core cable is still significantly cheaper than 24 cores. A 1 core cable
is the same price as 4 cores however.

On top of that, the price to splice is also linearly related to the number
of cores to splice. Yes there is the setup time, but then working on 192
cable takes a whole day, requires larger enclosures, requires larger
manholes, while we might only need 2 (!) splices to do the same work with
GPON.

Then there is the price to the ducting. A 192 core cable requires bigger
ducts and plastic is not only expensive, it has recently become scarce.
Putting in a 24 core cable in a 10/6 duct is much cheaper than a 192 core
cable.


> Then, PON, needing large closures for splitters and lengthy drop
> cables from the closures, costs a lot cancelling small cost of
> using dedicated cores of single star.
>


Now a splitter can be mounted in a splice enclosure taking up the same
space as 12 splices. We use dome shaped water tight enclosures for 96
splices and then we replace one of the splicing trays with the splitters.
All of this fits in a handhole about 70 cm long, 60 cm wide and 30 cm deep.

Another operator here instead has the splitters in cabinets with a cabinet
for every 50 to 200 passed homes. You could build a P2P network like that,
but then you would need power and active equipment in these cabinets.

Not sure what you are talking about with regards to drop cables. The house
connection is identical in a GPON and P2P network.



> On the other hand, if PON is assumed and the number of cores in a
> cable is small, core cost for single star will be large and only
> one PON operator with the largest share (shortest drop cable from
> closures to, e.g. 8 customers) can survive, resulting in monopoly.
>


Typically the infrastructure owner runs the PON equipment and resell vlan
based access to ISPs.

Regards,

Baldur

Re: Muni broadband sucks (was: New minimum speed for US broadband connections)

[Baldur Norddahl]

On Thu, Jun 3, 2021 at 10:44 AM Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Jim Troutman wrote:
>
> Private fiber operators are strongly motivated to deploy PON
> because PON is designed to make competitions impossible even
> if regulators forces the operators to do so, which is why
> PON is so popular.
>
> Muni fiber operators deploying PON because it is so pupular
> are just dumb stupid.
>


As the founder/owner of a private FTTH operator I can say the above is
wrong. The _only_ reason we use PON is because it is vastly cheaper to
build. It is also more flexible, which might be counter intuitive. I have
watched competitors try P2P but it is always a disaster for them. The PON
network will finish sooner, require considerably less cabling and ducts,
easier to expand with unplanned capacity, can be rerouted when an expected
permit fails to go through, and does not require much footprint for active
equipment. We have a single road side cabinet, using less than a single
square meter, serving an area in excess of 100 square kilometers. In theory
GPON can go all the way to 40 km from switch to customer, which would be
more than 1000 square km served from one point of presence.

Fiberstrands are not free. In a P2P topology you need to have cabinets with
active equipment close to the customers, otherwise you will have huge costs
for all that fiber. Your network would also become vulnerable because a
fiber cut on a duct with thousands of fiberstrands is not something that
gets fixed in a few hours. Huge cables can not easily be rerouted when
other construction works require you to do so.

Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On Thu, Jun 3, 2021 at 12:47 AM Seth Mattinen  wrote:

> UBNT's AirMax line is not "wifi". Their LTU line isn't either.
>
> Mike and Josh are actual WISP operators. You've stated you have no WISP
> experience. Listen to them.
>


Neither will listen to me when it comes to FTTH so nah :-)

Seriously, it appears to me that both are speaking from a legacy point of
view. The equipment deployed does neither use the new frequencies available
now, nor OFDMA which is a game changer. If nothing changes, 5G will beat
their pants off hands down.

Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

The kind of WISP we have around here is one or more AP on a tower or corn
silo and that one tower will cover a huge area by line of sight. There will
be nothing like you describe as each AP has separate frequency and
therefore no conflict. The gear is more or less standard wifi, often
Ubiquity.

If the density becomes great enough for scalability to be an issue, you
have a business case for fiber.

802.11ax has options for longer guard intervals to make it work at greater
distances.

On Wed, Jun 2, 2021 at 9:33 PM Mike Hammett  wrote:

> To have any sort of scalability, you take the free-for-all CSMA/CA and
> split it into uplink\downlink TDMA time slots. All APs transmit at the same
> time, then all APs listen at the same time.
>
> You then need to have the same uplink\downlink ratio on all APs in the
> system. To change the regulatory dynamics of upload\download then requires
> reconfiguration of the whole ecosystem to facilitate that, resulting in
> wasted cycles.
>
>
> BTW: A lot of WISPs use heavily modified versions of WiFi, but a lot also
> use platforms that have nothing in common with WiFi. Very, very few use
> straight 802.11. Why? Because it sucks at scale.
>
>
>
> Also, the extension of 802.11ax into the 6 GHz band will have variable
> results. Your usage is still a second class citizen (as it should be) to
> licensed users of the band.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Baldur Norddahl" 
> *To: *"NANOG" 
> *Sent: *Wednesday, June 2, 2021 11:07:45 AM
> *Subject: *Re: New minimum speed for US broadband connections
>
>
>
> tir. 1. jun. 2021 23.57 skrev Mike Hammett :
>
>>
>> Requiring a 100 meg upload really changes up the dynamics of the WISP
>> capabilities, resulting in fiber-only at a cost increase of 20x - 40x...
>>  for something that isn't needed.
>>
>
> I will admit to zero WISP experience but wifi is symmetrical speed up/down
> so why wouldn't a WISP not also be?
>
> Wifi 6E higher speed and base control of clients, subchannels,
> simultaneously transmission from multiple clients etc. All good stuff that
> should allow a WISP to deliver much higher upload.
>
> As soon a certain threshold is reached, higher speed will not cause more
> utilisation of the airwaves.
>
> The WISP will need to invest in wifi 6E gear, which I suspect is the real
> problem.
>
> Regards
>
> Baldur
>
>
>

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On Wed, Jun 2, 2021 at 7:05 PM Josh Luthman 
wrote:

> WISP is not symmetrical.  Wireless isn't symmetrical.  Nor is cable/dsl.
>

DSL splits the available frequencies into downstream and upstream, such
that usually much more frequencies are allocated downstream.  Wifi on the
other hand does no such thing. The clients and the base are exactly the
same and will send at the same bitrate. With wifi you can send or you can
receive, but not both at the same time. Which means wireless is perfectly
symmetrical in that you can either download at full speed, or upload at
full speed, but not both at the same time.


>
> WiFi 6E should have MU-MIMO which is something the WISPs have had for a
> few years, but not on equipment that speaks 802.11 WiFi.  That protocol
> wasn't really designed to do 1-15 miles, it was designed for 1-150 feet.
> That doesn't really have anything to do with upload, I don't know where you
> got that.
>
>
It is true that wifi is designed for short distances. That has not stopped
WISPs from using it for longer distances anyway.

Wifi 6E (802.11ax) has centrally controlled OFDMA which is used to assign
resource units to clients. This is completely different from previous wifi
versions. It means the selected frequency is split into 26 to 996 smaller
frequency bands, which can then be allocated to clients as needed. This
allows clients to send without any risk of collision with other clients and
clients can dynamically ask the base for more resource units, if it needs
to send much data etc. All of this is more like 5G than previous wifi. For
a WISP it should result in drastic improvements to upload, since you will
have less collisions and multiple clients sending at the same time.



> >As soon a certain threshold is reached, higher speed will not cause more
> utilisation of the airwaves.
>
> That's simply not going to happen.  Do you think the cell companies
> stopped deploying towers, too?
>

I am not sure what you want to say with the comment about cell companies. I
am saying that providing 10, 100 or 1000 Mbps upload to my customers on our
FTTH network makes little to no difference in the amount of upload that
happens. It will be the same on a WISP - why would it not be? So when you
go from dog slow upload to super fast upload, all that means is that the
airwaves will be idle more percent of the time. And when you do have the
occasional customer that uploads a lot, he will not step as much on the
other customers due to OFDMA.

Since Wifi shares the same frequencies for up and down, having one fast
will free time slots for the other.

Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

tir. 1. jun. 2021 23.57 skrev Mike Hammett :

>
> Requiring a 100 meg upload really changes up the dynamics of the WISP
> capabilities, resulting in fiber-only at a cost increase of 20x - 40x...
>  for something that isn't needed.
>

I will admit to zero WISP experience but wifi is symmetrical speed up/down
so why wouldn't a WISP not also be?

Wifi 6E higher speed and base control of clients, subchannels,
simultaneously transmission from multiple clients etc. All good stuff that
should allow a WISP to deliver much higher upload.

As soon a certain threshold is reached, higher speed will not cause more
utilisation of the airwaves.

The WISP will need to invest in wifi 6E gear, which I suspect is the real
problem.

Regards

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On Tue, Jun 1, 2021 at 2:27 AM Mike Hammett  wrote:

> No one's paying me anything except 15 years of practical experience
> building last mile networks for myself and my clients. I'd imagine that
> while a larger percentage than most venues, a minority of the people on
> this list build last mile networks. Even fewer do so with their own money.
>
> I have a fiber network where I offer gigabit bidirectional to the home.
>
>
> Few people have any sort of grasp of the cost and complexity of building
> what they want.
>
> Raising the the minimal definitions for everyone to what power users
> expect is a foolish venture.
>


Since you also replied to some of my comments, I will say that I am the
founder of a last mile FTTH provider in the greater Copenhagen, Denmark
area with thousands of customers. All built for our own money with zero
subsidies to customers that would pay good money to upgrade from DSL. I
planned, designed and built everything from the network, the outdoor plant,
the method we use to dig (directional drilling mostly), which pipes to use,
what cable etc. Also marketing, sales and funds raising - in short:
everything. We did this from nothing to a company with more than 100
employees today.

I claim to know the cost and complexity better than most.

I'm just trying to connect some of you to reality.
>
>
I could say the same. But maybe our reality differs. You seem to be very
hung up on what minimums are needed to do a certain job. But that simply is
not it. If a person believes his internet is slow, then it is slow, no
matter what some experts think would be enough for that persons needs. That
means he will buy my offering even though he probably already has VDSL with
speeds faster than what you propose. It also means he will consider the
available options when weighting pros and cons of a new home.

Here in Denmark we have a problem that people are moving away from rural
areas and to the bigger cities. There are many reasons for this, but one
often quoted reason is the lack of good internet.

Good internet in Denmark is 1000 Mbps for less than USD $50 per month. But
I accept that 100 Mbps at a somewhat higher price point is probably a fine
speed for rural US, where distances are huge and alternative solutions,
such as fixed wireless, may need to be part of the solution. Or maybe
Starlink is the solution.

Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

On 31.05.2021 06.52, Mark Tinka wrote:



On 5/29/21 00:38, Lady Benjamin Cannon of Glencoe, ASCE wrote:


8 billion fiber drops for 8 billion people.


Technically speaking, 8 billion people is not 8 billion households :-).

But the bigger problem is getting fibre to every family in the world 
is not yet currently feasible.


There is a reason developing markets have a lot more mobile phones 
than they have people, or the energy to charge them.



But why would the goal be fiber to every household? There are other ways 
to deliver good internet. In fact all of the major platforms can do so: 
fiber, coax, DSL, fixed wireless, 4G / 5G. The fiber platform will do so 
naturally, the others may require some extra investment but are still 
options.


Of course there are developing countries where the goal is any internet 
at all. I hope that is not the case for US broadband.


Regards,

Baldur

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

søn. 30. maj 2021 15.29 skrev Mike Hammett :

> What can you do with 100 megs that you can't do with 25 megs and why
> should anyone care?
>

That is really the wrong question. People want 100 Mbps over 25 Mbps and
therefore it becomes a need for rural communities. Doesn't matter that
someone believes these people could do with less.

The year is 2021 and perceived good internet is minimum 100 Mbps.

Regards

Baldur


>

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

In 2021 I would claim that 100 Mbps is where "good" internet starts. Yes,
25 Mbps will work but it is not good internet. Not to mention 2 Mbps ADSL
which is almost the same as no internet.

Now there are needs for an individual and there are needs for a community.
The rural communities have a genuine need for good internet. Anything less
will continue to accelerate the move of people to the cities. In the end,
each individual decides for himself what his needs are and sufficient
people want good internet, so that they will have that as part of their
considerations when deciding to move.

With poor internet the community will accumulate people that dont care
about the internet, which often means elderly people. More elderly people
means younger people do not feel at home, so they will move away or not
move there, which further accelerates the effect.

Regards,

Baldur


On Sat, May 29, 2021 at 11:19 PM Andy Ringsmuth  wrote:

> Well, honestly, if you really want to go down the “need vs. want” road,
> 100 percent of the folks on this list would be out of a job.
>
> What are genuine needs? Food/water, clothing and shelter. That’s it. Even
> the last two are somewhat negotiable if you get right down to it.
>
> 
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> a...@andyring.com
>
> “Better even die free, than to live slaves.” - Frederick Douglas, 1863
>
> > On May 29, 2021, at 7:48 AM, Mike Hammett  wrote:
> >
> > Need vs. want.
> >
> >
> >
> > -
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> > Midwest-IX
> > http://www.midwest-ix.com
> >
> > From: "Baldur Norddahl" 
> > To: "NANOG" 
> > Sent: Saturday, May 29, 2021 3:49:01 AM
> > Subject: Re: New minimum speed for US broadband connections
> >
> > I am in Europe / Denmark. The EU has defined broadband to be 100 Mbps
> download with nothing specified for upload. The goal is for everyone to
> have access to broadband by 2025.
> >
> > Such definitions do help those in rural areas. In fact this is precisely
> useful for those that do not currently have access. It helps to make goals
> and to measure how we are progressing.
> >
> > All current technologies can deliver broadband, including DSL, coax, 5G
> and fixed wireless. But maybe not without investment. That DSL plant might
> need upgrading to the latest VDSL and cabinets closer to the customer. The
> coax might need upgrades etc. But that is the point. Providers will need to
> invest to be able to claim broadband.
> >
> > On the other hand a soft easy broadband definition is useless in my
> opinion. Then everyone has broadband, hurray, but many have slow internet
> and nothing is going to be done because it is broadband!
> >
> > Regards
> >
> > Baldur
>
>

Re: New minimum speed for US broadband connections

[Baldur Norddahl]

I am in Europe / Denmark. The EU has defined broadband to be 100 Mbps
download with nothing specified for upload. The goal is for everyone to
have access to broadband by 2025.

Such definitions do help those in rural areas. In fact this is precisely
useful for those that do not currently have access. It helps to make goals
and to measure how we are progressing.

All current technologies can deliver broadband, including DSL, coax, 5G and
fixed wireless. But maybe not without investment. That DSL plant might need
upgrading to the latest VDSL and cabinets closer to the customer. The coax
might need upgrades etc. But that is the point. Providers will need to
invest to be able to claim broadband.

On the other hand a soft easy broadband definition is useless in my
opinion. Then everyone has broadband, hurray, but many have slow internet
and nothing is going to be done because it is broadband!

Regards

Baldur

Re: BGP Traffic Engineering - Active\Passive

[Baldur Norddahl]

Hello

First one needs to remember that it is always the sender that ultimately
decides which path to use. You can use route-map or import policy to
override local pref for each matched received prefix to steer exactly which
ISP you want to use on a per prefix basis. But so can everyone else.

Say you are AS1 and we are AS2. We have two common IP transit providers
AS100 and AS200. You really want to use AS100 so you will prepend on AS200.
However AS2 is free to ignore that prepend and set a lower local pref for
your prefix towards AS200. Or the most obvious example, AS2 could simply be
using AS200 as default, so nothing at all could make them use AS100 for
sending.

Now say AS100 and AS200 are actually connected. AS2 delivers the packet to
AS200. AS200 sees that you prepended your prefix on your direct connection
to AS200. The path length is much shorter if they deliver the packet to
AS100 instead of delivering it directly to you due to prepending. In 99% of
the cases, a transit provider will ignore path length in this situation.
Everyone has a rule to deliver the packet directly to a customer if that is
possible. Just one of many examples where path length is ignored in
practical BGP.

The only thing that generally works 100% is announcing a more specific
prefix through one transit provider. That will however move everything to
that provider leaving zero traffic on the other providers.

Some transit providers allow you to use communities to selectively prepend
towards certain destinations as described by others here. Just remember
that you really do not have all that much control over the ingress
direction. Not to say it does not work, but it may disappoint somewhat.

Regards,

Baldur

DDoS attack with blackmail

[Baldur Norddahl]

Hello

We got attacked by a group that calls themselves "Fancy Lazarus". They want
payment in BC to not attack us again. The attack was a volume attack to our
DNS and URL fetch from our webserver.

I am interested in any experience in fighting back against these guys.

Thanks,

Baldur

Re: Juniper hardware recommendation

[Baldur Norddahl]

man. 10. maj 2021 16.20 skrev :

> I prefer MX204 over the ACX5048.  The ACX5048 can’t add L3 interface to an
> mpls layer 2 type of service.  There are other limitations to the ACX5048
> that cause me to want to possibly replace them with MX204’s.  But in
> defense of the ACX5048, we have gotten some good mileage (a few years now)
> of good resi/busi bb over vrf’s and also carrier ethernet for businesses
> and lots of cell backhaul… so they are good for that.  I’ve heard the
> ACX5448 was even better.
>

It is my understanding that acx5448 is much more capable than the older
acx5048. It will definitely do both l2vpn and l3vpn on mpls (what we use
acx5448 / acx710 for).

The main limitation is that it will not do full dfz table and not more
exotic stuff like subscriber management.

Regards

Baldur

Re: Juniper hardware recommendation

[Baldur Norddahl]

lør. 8. maj 2021 22.56 skrev Mark Tinka :

>
>
> On 5/8/21 22:50, Baldur Norddahl wrote:
>
> >
> > Maybe they did in the ACX710? Does most things except full routing table.
>
> We looked at it. Apart from supporting only DC power (which we don't
> like), it's Broadcom.
>
> Granted, there's a whole new line of ACX7XXX boxes they are putting out,
> one of which we shall be testing. So I'm giving Broadcom a chance.
>
> Mark.
>

It is possible to get a 48V 6A DC power supply as a power brick laptop
style. Just look at it as an external psu :-)

I will admit that we use them with 48 volt battery banks the way intended.
It has become extremely cheap and easy to make your own with lithium iron
phosphate batteries.

Regards

Baldur

Re: Juniper hardware recommendation

[Baldur Norddahl]

lør. 8. maj 2021 09.16 skrev Mark Tinka :

>
>  I just wish Juniper could make an MX204-lite, one with more 10Gbps port
> density, e.t.c.
>

Maybe they did in the ACX710? Does most things except full routing table.

We use mx204 to carry the full tables and handle ip transit. And ACX5448 +
ACX710 with evpn, vpls, l3vpn and internal Internet tables.

Regards

Baldur

link monitoring

[Baldur Norddahl]

Hello

We had a 100G link that started to misbehave and caused the customers to
notice bad packet loss. The optical values are just fine but we had packet
loss and latency. Interface shows FEC errors on one end and carrier
transitions on the other end. But otherwise the link would stay up and our
monitor system completely failed to warn about the failure. Had to find the
bad link by traceroute (mtr) and observe where packet loss started.

The link was between a Juniper MX204 and Juniper ACX5448. Link length 2
meters using 2 km single mode SFP modules.

What is the best practice to monitor links to avoid this scenarium? What
options do we have to do link monitoring? I am investigating BFD but I am
unsure if that would have helped the situation.

Thanks,

Baldur

Re: 10g residential CPE

[Baldur Norddahl]

On Mon, Dec 28, 2020 at 8:48 PM Seth Mattinen  wrote:

> On 12/28/20 9:11 AM, Aaron Wendel wrote:
> > Actually our free service doesn't have limitations, has an SLA, no
> > time/term restrictions, a CPE, support, etc.
>
>
> How do SLA refunds work on free service? Do you just pay them some cash
> value instead of credits?
>

I find SLA refunds are meaningless anyway. The SLA is more about stating
what level of service is expected. Then we can tell if we succeeded or
failed in delivering what was expected. In the case of failure nothing can
fix that other than a plan for how it can be improved going forward.
Getting money back will usually not do much to fix the hardship poor
service put you through.

Regards,

Baldur

Re: 10g residential CPE

[Baldur Norddahl]

I applaud your commitment to helping your local community. Just want to
point out that this is a charity because it does not scale. Nobody could
build out a FTTH network and make it free as a business case. But there are
plenty of people that made a network for their neighbors and provided that
for free. Maybe a person had a commercial fiber to his home and thought he
could just as well share it. This might be on a bigger scale but it is the
same.

Regards,

Baldur


On Mon, Dec 28, 2020 at 8:27 PM Aaron Wendel 
wrote:

> Darin,
>
> Our business support and residential support is the same department.  I
> have to pay those people to be in the office either way so it doesn't
> cost me any "more" to provide support for the residences. Yes, walking
> Grandma through getting her email can sometimes be a chore but that
> person is on the payroll whether he/she is helping Grandma or sitting
> there chatting with his/her co-worker.  If we dumped all the residential
> customers we would still have the same cost structure we do now.
>
> Again, it's been free for the last 7 years at this point.  I've never
> been one to really do what I "should" anyway.
>
> Aaron
>
>
> On 12/28/2020 11:48 AM, Darin Steffl wrote:
> > Aaron,
> >
> > The "Free" service doesn't cover your cost of support which is much
> > higher for residential than any business customer. Our residential
> > customers call at least 15x more often compared to business customers
> > compared on a 1:1 ratio.
> >
> > I honestly can't fathom providing free residential service because we
> > make enough money on the business side of things. You should be
> > charging something, at least $20-30 per month.
> >
> > On Mon, Dec 28, 2020 at 11:15 AM Aaron Wendel
> > mailto:aa...@wholesaleinternet.net>>
> wrote:
> >
> > The $300 covers the equipment and the time to send someone out to a
> > house to install it.  If $300 is too much you can pay in 12
> > installments
> > of $25.
> >
> > The TIK alone costs us about $250.
> >
> > Aaron
> >
> >
> > On 12/27/2020 5:04 AM, Mark Tinka wrote:
> > >
> > >
> > > On 12/26/20 20:48, Darin Steffl wrote:
> > >
> > >> Aaron,
> > >>
> > >> One simple question. Why on earth would you offer free internet
> > >> service? How and why? Your site show 1 Gig symmetrical for free
> > when
> > >> you should be a minimum of $65 per month to be competitive.
> > >
> > > They also ask for no monthly fee after a single payment of US$300.
> > >
> > > Considering the 2Gbps package costs US$49.95, you'd guess they'd
> > value
> > > the 1Gbps service at, say US$27/month, give or take.
> > >
> > > So that US$300 provides a bit of coverage, perhaps 1 year, in which
> > > time they'd have likely upgraded the customer.
> > >
> > > Mark.
> >
> > --
> > 
> > Aaron Wendel
> > Chief Technical Officer
> > Wholesale Internet, Inc. (AS 32097)
> > (816)550-9030
> > http://www.wholesaleinternet.com 
> > 
> >
> >
> >
> > --
> > Darin Steffl
> > Minnesota WiFi
> > www.mnwifi.com 
> > 507-634-WiFi
> > Like us on Facebook 
>
> --
> 
> Aaron Wendel
> Chief Technical Officer
> Wholesale Internet, Inc. (AS 32097)
> (816)550-9030
> http://www.wholesaleinternet.com
> 
>
>

Re: [External] Re: 10g residential CPE

[Baldur Norddahl]

søn. 27. dec. 2020 19.00 skrev Valdis Klētnieks :

> On Sun, 27 Dec 2020 17:57:17 +0100, Baldur Norddahl said:
>
> > Here in the civilised world we bury the wires ;-)
>
> Even the long-haul 765kv and up connections across the power grid?
>
> In the US, they're out on towers for a reason - you can fly along them
> in a helicopter and easily spot parts of cable that are degrading and need
> repair because they glow brighter on an infrared scope...
>
> (Plus, as Hurricane Sandy taught Manhattan, buried wires have their
> own rather nasty failure modes)
>

All of the 400V and 10 kV is buried. That means no wires along streets,
anywhere.

The long haul transmission network consists mostly of 150 kV and 400 kV
lines. That has been partly buried, especially near and in cities. There
was a project to have it all buried but was abandoned halfway due to cost.

But then it is all fully redundant, so they will just power it down if it
needs maintenance. My company is digging for FTTH and in the few cases we
need to cross one of these bad guys, they will shut it for us while we are
working. Nobody looses power of course.

The 10 kV network is redundant too. We managed to hit those a few times.
That will cause a power interruption for 10 to 20 minutes until they
reroute the power. I believe mostly for safety, they need to be sure that
the damaged line will not become energized again.

Regards

Baldur

Re: [External] Re: 10g residential CPE

[Baldur Norddahl]

søn. 27. dec. 2020 17.14 skrev Michael Thomas :

>
>
> We have both, and are going to get a battery. But the battery would
> probably only be good for about a day which is not enough, especially
> with these planned shutoffs because they have to inspect their wire
> plant in daylight. There has to be a better technical solution for this
> beyond just burying the wires. A properly trained AI could probably
> figure out what's naught and nice.
>

Here in the civilised world we bury the wires ;-)

Re: 10g residential CPE

[Baldur Norddahl]

It was not meant to be a test as such, just a demonstration. Netnod to
Bahnhof is full speed and the third server is mine, so all three servers
can deliver at least 1G.

Finding a speedtest.net server at least 1000 km away that will show full
speed at 1G is hard. Namely because most such servers have at least 10G NIC
and that is not an advantage.

It is possible to get 1G at 10 ms, I did demonstrate that myself with the
test to Bahnhof. It is also possible to be limited at 30%. As the test to
Netnod shows.


lør. 26. dec. 2020 20.10 skrev Filip Hruska :

> I wouldn't rely on these numbers too much, your testing methodology is
> flawed.
> People don't expect RING nodes to be used as speedtest servers and so they
> are usually not connected to high speed networks.
>
> Using a classical speedtest.net (Web or CLI) application would make much
> more sense, given the servers are actually connected to high speed Internet
> and are tuned to achieve such speeds - which is much more akin to how the
> most bandwidth demanding stuff (streaming, game downloads, system updates
> from CDNs) behaves.
>
> It's certainly possible to get 1G+ over >10ms RTT connections single
> stream - the buffers are certainly not THAT small for it to be a problem -
> not to mention game distribution platforms do usually open multiple
> connections to maximise the bandwidth utilisation.
>
> Re 85KB: that's just the initial window size, which will grow given tcp
> window scaling is enabled (default on modern Linux).
>
> Filip
>
>
> On 26 December 2020 19:14:13 CET, Baldur Norddahl <
> baldur.nordd...@gmail.com> wrote:
>>
>>
>>
>> lør. 26. dec. 2020 18.55 skrev Mikael Abrahamsson :
>>
>>> On Sat, 26 Dec 2020, Baldur Norddahl wrote:
>>>
>>> > It is true there have been TCP improvements but you can very easily
>>> verify
>>> > for yourself that it is very hard to get anywhere near 1 Gbps of actual
>>> > transfer speed to destinations just 10 ms away. Try the nlnog ring
>>> network
>>> > like this:
>>> >
>>> > gigabit@gigabit01:~$ iperf -c netnod01.ring.nlnog.net
>>> > 
>>> > Client connecting to netnod01.ring.nlnog.net, TCP port 5001
>>> > TCP window size: 85.0 KByte (default)
>>> > 
>>> > [  3] local 185.24.168.23 port 50632 connected with 185.42.136.5 port
>>> 5001
>>> > [ ID] Interval   Transfer Bandwidth
>>> > [  3]  0.0-10.0 sec   452 MBytes   379 Mbits/sec
>>>
>>> Why would you just use 85KB of TCP window size?
>>>
>>> That's not the problem of buffering (or lack thereof) along the path,
>>> that
>>> just not enough TCP window size for long-RTT high speed transfers.
>>>
>>
>> That is just the starting window size. Also it is the default and I am
>> not going to tune the connection because no such tuning will occur when you
>> do your next far away download and wonder why it is so slow.
>>
>> If you do the math you will realise that 379 Mbps at 10 ms is impossible
>> with 85 K window.
>>
>> I demonstrated that it is about buffers by showing the same download from
>> a server that paces the traffic indeed gets the full 930 Mbps with exactly
>> the same settings, including starting window size, and the same path
>> (Copenhagen to Stockholm).
>>
>> Regards
>>
>> Baldur
>>
>>
>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

Re: 10g residential CPE

[Baldur Norddahl]

On Sat, Dec 26, 2020 at 7:28 PM Mikael Abrahamsson  wrote:

> On Sat, 26 Dec 2020, Baldur Norddahl wrote:
>
> > I demonstrated that it is about buffers by showing the same download
> > from a server that paces the traffic indeed gets the full 930 Mbps with
> > exactly the same settings, including starting window size, and the same
> > path (Copenhagen to Stockholm).
>
> You demonstrated that it's about which TCP algorithm they use, probably.
>
>
All (virtual) machines used in the experiment are the same. Those are NLNOG
RING network managed machines all running the exact same Ubuntu 16.04.7 LTS.

If you have access to NLNOG RING or equivalent you should try the
experiment for yourself. You will find that as latency increases TCP speeds
goes down and it can not be explained by congestion. And you will find that
some servers have this effect much less than others and that those servers
usually have 1G network speed. The effect is the same no matter what time
of day you try it (ie. it is not congestion related).

Before you panic I will say I am not trying to advocate that we need more
buffers. We need smart buffers. Buffer bloat is bad but no buffers is also
bad. Your home made debloat solution will probably not be able to recover
the missing TCP performance that I am describing here. But if you could
have FQ Codel in the ISP switch that would probably do a lot.

Or we could have TCP with pacing and that will be widely deployed around
the same time as IPv6.

Regards

Baldur

Re: 10g residential CPE

[Baldur Norddahl]

lør. 26. dec. 2020 18.55 skrev Mikael Abrahamsson :

> On Sat, 26 Dec 2020, Baldur Norddahl wrote:
>
> > It is true there have been TCP improvements but you can very easily
> verify
> > for yourself that it is very hard to get anywhere near 1 Gbps of actual
> > transfer speed to destinations just 10 ms away. Try the nlnog ring
> network
> > like this:
> >
> > gigabit@gigabit01:~$ iperf -c netnod01.ring.nlnog.net
> > 
> > Client connecting to netnod01.ring.nlnog.net, TCP port 5001
> > TCP window size: 85.0 KByte (default)
> > 
> > [  3] local 185.24.168.23 port 50632 connected with 185.42.136.5 port
> 5001
> > [ ID] Interval   Transfer Bandwidth
> > [  3]  0.0-10.0 sec   452 MBytes   379 Mbits/sec
>
> Why would you just use 85KB of TCP window size?
>
> That's not the problem of buffering (or lack thereof) along the path, that
> just not enough TCP window size for long-RTT high speed transfers.
>

That is just the starting window size. Also it is the default and I am not
going to tune the connection because no such tuning will occur when you do
your next far away download and wonder why it is so slow.

If you do the math you will realise that 379 Mbps at 10 ms is impossible
with 85 K window.

I demonstrated that it is about buffers by showing the same download from a
server that paces the traffic indeed gets the full 930 Mbps with exactly
the same settings, including starting window size, and the same path
(Copenhagen to Stockholm).

Regards

Baldur

Re: 10g residential CPE

[Baldur Norddahl]

On Sat, Dec 26, 2020 at 5:41 PM Mikael Abrahamsson  wrote:

> On Sat, 26 Dec 2020, Baldur Norddahl wrote:
>
> > That is why. The RTT to the source can not be larger than the minimum
> > buffer size in the transport path. Otherwise the speed will start
> > decreasing.
>
> This is no longer correct. There has been lots of TCP innovation since
> this was true.
>
> Please stop repeating it.
>

It is true there have been TCP improvements but you can very easily verify
for yourself that it is very hard to get anywhere near 1 Gbps of actual
transfer speed to destinations just 10 ms away. Try the nlnog ring network
like this:

gigabit@gigabit01:~$ iperf -c netnod01.ring.nlnog.net

Client connecting to netnod01.ring.nlnog.net, TCP port 5001
TCP window size: 85.0 KByte (default)

[  3] local 185.24.168.23 port 50632 connected with 185.42.136.5 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec   452 MBytes   379 Mbits/sec

And that is a direct peer of ours.

In general you will have trouble with any server that has a NIC > 1G. If
you find a server that has a 1G NIC this happens instead:

gigabit@gigabit01:~$ iperf -c bahnhof01.ring.nlnog.net

Client connecting to bahnhof01.ring.nlnog.net, TCP port 5001
TCP window size: 85.0 KByte (default)

[  3] local 185.24.168.23 port 56412 connected with 195.178.185.171 port
5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec  1.08 GBytes   930 Mbits/sec

Why? Because the 1G NIC server naturally will pace the traffic at maximum
1G and therefore not fill any buffers in the transfer path. The 10G servers
on the other hand WILL fill the buffers and experience packet loss.

Regards,

Baldur

Re: 10g residential CPE

[Baldur Norddahl]

lør. 26. dec. 2020 16.35 skrev Mikael Abrahamsson via NANOG :

>
>
>
> Perhaps there are some issues at other parts of the network that limits
> their speeds? I'm in Stockholm, Sweden, with plenty of local CDNs located
> just 1-3ms away from me.
>

That is why. The RTT to the source can not be larger than the minimum
buffer size in the transport path. Otherwise the speed will start
decreasing.

Since a lot of ISP equipment only has tiny buffers you will generally be
unable to get great downloads from sources far away.

The other option is huge buffers which gives you better speed but with
buffer bloat issues.

Regards

Re: [External] Re: 10g residential CPE

[Baldur Norddahl]

fre. 25. dec. 2020 21.49 skrev Michael Thomas :

>
> On 12/25/20 12:40 PM, Chris Adams wrote:
> >
> > The other aspect of it is that we're doing these downloads while
> > continuing to play other games and chat (both things sensitive to
> > latency).  Some have family/roommates in the home, so they may be
> > streaming audio and/or video at the same time.  Do we fill up a gigabit?
> > No, probably not... but we'd notice if we had a lot less.
>
> But using the right queuing disciplines it a lot cheaper than the brute
> force and ignorance of just upping the bandwidth, right?
>
> It seems really surprising after almost a decade of discovery of
> bufferbloat that most CPE are still doing tail drops.
>
> Mike
>

For download that queue discipline needs to be implemented at the ISP end.
It is just a week or so since I asked what other operators were doing with
that and I got very few replies. So maybe we can assume the answer is not
much.

I also learned that the big iron providers J and C only implements tail
drop and WRED. That's it. It is not sufficient to provide good service, so
the only option is to throw more bandwidth at the problem.

If the operator wants to keep bufferbloat low you will not be able to
utilise your 1 Gbps to that speed when downloading from distant servers.
But with the same bufferbloat measured in milliseconds you will still have
a 10x bigger buffer and thus 10x bigger bandwidth delay product. That
translates to 10x the speed. That speed might just be 100 Mbps on your 1000
Mbps connection. But it would have been just 10 Mbps on a 100 Mbps...

Regards

Baldur

>

best current practice: buffers

[Baldur Norddahl]

Hello

What is the best current practice for buffer size? For customer facing
ports, core network ports and transit links?

We have a buffer problem, discovered by a customer that moved their servers
to a cloud service some distance away. That resulted in a drastic reduced
transfer speed between their office and the cloud service. Nothing much
could be done since we, like so many others, have switches with extreme
fast port speeds (48x 10G, 4x 100G) with a tiny shared 12 MB buffer.

Now the time has come to upgrade that hardware to something that does have
plenty of buffer capacity, so I am planning out what the settings should be.

I have read this paper
http://web.stanford.edu/class/cs244/papers/sizing-router-buffers-redux.pdf
which claims not much buffer is needed at all. And I think they are
completely wrong. In the paper they assume we are trying to get as much
throughput on a congested core or transit port. But we always make sure
those are not congested, so that misses the mark completely. For core ports
we are concerned about microbursts and for customer ports we do care about
a single TCP session being able to get the max throughput. The paper
assumes there will be a lot of TCP sessions sharing the bandwidth, but that
is not always the case with customer ports. It might be a guy downloading
an ISO image with him being the only guy at the office.

The common wisdom is to set the buffer size to one bandwidth-delay product.
And also that bigger buffers than this is harmful. But that raises the
question of what distance do I tune for? Amsterdam is 10 ms away. East
coast USA is 100 ms.

Also the new hardware (Juniper ACX710) does support more than one queue per
port. Would it be possible to have that ISO download go into a queue for
heavy streams and allow smaller streams to skip the line, so we do not see
the downside of a heavy buffer (buffer bloat)? It is no longer just a
simple matter of buffer size.

What are others doing to deliver the best possible performance to customers
with regards to buffering?

Regards,

Baldur

urpf - evil?

[Baldur Norddahl]

Hello

While working on my ACLs I noticed that I was successful in blocking some
apparently spoofed IPv6 traffic. The destination was Facebook and the
source was IPv6 range belonging to a mobile operator that sells 4G Wifi
router based solutions.

So thinking about how and why a few customers end up sending packets to our
network with the wrong source, I came up with a theory (not validated):
What if the customer connects his 4G Wifi router to one of the LAN ports of
our CPE (or visa versa)? His computer would then pick up an IPv6 range from
both ISPs along with two default routes. But only one default route would
be used, and in this case that was apparently the default route going to
our network. But still his computer might use the IPv6 address from the
other ISP as source and therefore he ends up "spoofing" by sending that to
us. We deliver the packets to Facebook and I assume Facebook will route the
replies just fine through the other ISP.

Now the thing is that my impression is that it actually works so long I do
not actively block it with uRPF or ACLs on our edge. I have learned that
spoofing is evil and I should be blocking this - but why am I sabotaging
something that apparently is doing just fine at some customers?

Regards,

Baldur

Re: cheap MPLS router recommendations

[Baldur Norddahl]

Does this device have deep buffers?

On Wed, Oct 21, 2020 at 11:12 PM Colton Conor 
wrote:

> https://www.multicominc.com/wp-content/uploads/DZS-M3000_M.pdf
>
> On Wed, Oct 21, 2020 at 4:08 PM Colton Conor 
> wrote:
>
>> Well then Adam I would say the Dasan Zhone fits the budget. The M3000
>> seems like a real beast for the price point with 100G ports.
>>
>> Yes, other whitebox vendors are doing this, but they seem to want 2-4k
>> for the whitebox, and even more for the operating system, making it more
>> expensive that Juniper from what I have seen.
>>
>> On Wed, Oct 21, 2020 at 3:27 PM  wrote:
>>
>>> Just to clarify what cheap means, ideally  -$2000 to $4000 new
>>>
>>> -new is preferred as buying used kit on second hand market one is at the
>>> mercy of the price fluctuations and availability.
>>>
>>>
>>>
>>> And the likes of the M2400 looks good 4x10G plus some 1G, unfortunately
>>> there are no details on the webpage (and the datasheet can’t be downloaded…
>>> )
>>>
>>>
>>>
>>> Are there more folks out there bundling open NOS and white-box HW along
>>> with the support for the whole thing?
>>>
>>>
>>>
>>>
>>>
>>> adam
>>>
>>>
>>>
>>> *From:* NANOG  *On
>>> Behalf Of *Colton Conor
>>> *Sent:* Monday, October 19, 2020 4:51 PM
>>> *To:* t...@pelican.org
>>> *Cc:* NANOG 
>>> *Subject:* Re: cheap MPLS router recommendations
>>>
>>>
>>>
>>> I haven't tried one myself, but Dasan Zhone has the M2400 and M3000.
>>> Basically, a whitebox with IP Infusion code on it. New, I think the price
>>> point is sub $2000 to $4000 new. That's a ton of ports for that price
>>> point. Anyone tried these yet?
>>> https://dzsi.com/product-category/mobile-xhaul/
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Oct 19, 2020 at 3:38 AM t...@pelican.org  wrote:
>>>
>>> On Saturday, 17 October, 2020 00:41, "Tony Wicks" 
>>> said:
>>>
>>> > Well, there is always the MX104 (if you want redundancy) or MX80 if you
>>> > don’t. That will give you 80gig wire speed just don’t load it up with
>>> > more than one full table.
>>>
>>> Bear in mind that the MX80 is now in the EoL process, you have <4 years
>>> of support left.  Depending on your expected life-time / depreciation
>>> rules, buying one new right now might be unwise.
>>>
>>> Do *not* throw a full table at it (or any of the PowerPC Junipers)
>>> unless you have a lot of patience for reconvergence, and black-holes while
>>> you wait.
>>>
>>> MX104 is a nice box for getting dual-RE in something relatively compact
>>> and cheap, and has environmental hardening if that matters to you, but is
>>> still not best pleased with full tables.
>>>
>>> OP could do with clarifying "cheap" :)
>>>
>>> Regards,
>>> Tim.
>>>
>>>

Re: Ingress filtering on transits, peers, and IX ports

[Baldur Norddahl]

Might filtering port 11211 like that not risk blocking random connections,
where the operating system picked that port as source, which then becomes
destination on the reply packets?

tir. 20. okt. 2020 07.19 skrev Randy Bush :

> term blocked-ports {
> from {
> protocol [ tcp udp ];
> first-fragment;
> destination-port
> [ 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445
> syslog 11211];
> }
> then {
> sample;
> discard;
> }
> }
>
> and i block all external access to weak devices such as switches, pdus,
> ipmi, ...
>
> randy
>

Re: cheap MPLS router recommendations

[Baldur Norddahl]

Juniper ACX710. Yes it also has more ports, but you only pay for the
capacity you need (100G minimum). So you could buy a license that would
allow you to enable 10x 10G with the 100G ports dormant.

On Fri, Oct 16, 2020 at 11:57 PM  wrote:

> For this particular gig even the MX204 would be overkill in terms of price
> as well as performance.
>
> Ideally something like 204 but with only those 8 10/1G ports (i.e. without
> the 4x100G ports)
>
>
>
> adam
>
> *From:* Tony Wicks 
> *Sent:* Friday, October 16, 2020 10:36 PM
> *To:* adamv0...@netconsultings.com
> *Cc:* nanog@nanog.org
> *Subject:* RE: cheap MPLS router recommendations
>
>
>
> Juniper MX204, easy
>
>
>
> *From:* NANOG  *On Behalf Of *
> adamv0...@netconsultings.com
> *Sent:* Saturday, 17 October 2020 10:31 am
> *To:* 'Jakub Horn (jakuhorn)' ; nanog@nanog.org
> *Subject:* RE: cheap MPLS router recommendations
>
>
>
> Yeah the XR thing would be great but NCS540 would be too expensive and too
> much throughput meaning draws too much power,
>
>
>
> adam
>

Re: Ingress filtering on transits, peers, and IX ports

[Baldur Norddahl]

This is about ingress ACL not egress.

tor. 15. okt. 2020 12.00 skrev :

> Simple,
>
> All stub autonomous systems should have a simple egress ACL allowing only
> PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit
> points (towards transits/peers), that’s it.
>
> -not sure why this isn’t the first sentence in every BCP and “security
> bulletin”…
>
>
>
>
>
> adam
>
>
>
> *From:* NANOG  *On
> Behalf Of *Baldur Norddahl
> *Sent:* Thursday, October 15, 2020 8:38 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Ingress filtering on transits, peers, and IX ports
>
>
>
> All DNS resolvers discovered on our network belong to customers. Our own
> resolvers, running unbound, were not discovered.
>
>
>
> While filtering same AS on ingress could help those customers (but only
> one was a open relay), filtering bogons is something the customer can also
> do. Or the software can be fixed. Do we really expect the ISP to implement
> firewalls instead of customers upgrading software?
>
>
>
> I also note that apparently our own ISPs (transits) do not filter bogons
> either.
>
>
>
> The above is a principal question. I am going to filter bogons, it just is
> not very high on my long list of stuff to do.
>
>
>
> Regards
>
>
>
> Baldur
>
>
>
>
>
> ons. 14. okt. 2020 20.53 skrev Casey Deccio :
>
> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway  wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks.  Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey
>
>

Re: Ingress filtering on transits, peers, and IX ports

[Baldur Norddahl]

All DNS resolvers discovered on our network belong to customers. Our own
resolvers, running unbound, were not discovered.

While filtering same AS on ingress could help those customers (but only one
was a open relay), filtering bogons is something the customer can also do.
Or the software can be fixed. Do we really expect the ISP to implement
firewalls instead of customers upgrading software?

I also note that apparently our own ISPs (transits) do not filter bogons
either.

The above is a principal question. I am going to filter bogons, it just is
not very high on my long list of stuff to do.

Regards

Baldur


ons. 14. okt. 2020 20.53 skrev Casey Deccio :

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway  wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks.  Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

Re: Hurricane Electric AS6939

[Baldur Norddahl]

On Wed, Oct 14, 2020 at 1:30 AM Aaron Gould  wrote:

> Do y’all like HE for Internet uplink?  I’m thinking about using them for
> 100gig in Texas.  It would be for my eyeballs ISP.  We currently have
> Spectrum, Telia and Cogent.
>
> -Aaron
>

I find HE useful as a special kind of transit provider. They have more
peerings than anyone and they have peerings that you will not be able to
get yourself. They are not necessarily a replacement for connecting
directly to local internet exchanges, but rather a supplement.

Unless you can afford to have most of the top tier transit providers, you
will often get a more direct path through the many peerings of HE. Then as
you grow and establish your own peerings, you will move some of that
traffic away from HE to your own.

One gotcha is that for some unknown reason, some of the big content
networks do apparently not peer with HE. However as an eyeball network you
will have little trouble getting those directly.

Regards,

Baldur

Re: Securing Greenfield Service Provider Clients

[Baldur Norddahl]

Are you really suggesting decrypting customer traffic? In most parts of the
world that act falls in one of two categories: it is either required by law
or it is illegal.

Offer your customers a good virus scanner to install instead.

Regards

Baldur


fre. 9. okt. 2020 21.27 skrev Christopher J. Wolff :

> Dear Nanog;
>
>
>
> Hope everyone is getting ready for a good weekend.  I’m working on a
> greenfield service provider network and I’m running into a security
> challenge.  I hope the great minds here can help.
>
>
>
> Since the majority of traffic is SSL/TLS, encrypted malicious content can
> pass through even an “NGFW” device without detection and classification.
>
>
>
> Without setting up SSL encrypt/decrypt through a MITM setup and handing
> certificates out to every client, is there any other software/hardware that
> can perform DPI and/or ssl analysis so I can prevent encrypted malicious
> content from being downloaded to my users?
>
>
>
> Have experience with Palo and Firepower but even these need the MITM
> approach.  I appreciate any advice anyone can provide.
>
>
>
> Best,
>
> CJ
>

Re: BFD for routes learned trough Route-servers in IXPs

[Baldur Norddahl]

Hello

ARP timeout should be lower than MAC timeout, but usually the default is
the other way around. Which is extremely stupid. To those who do not know
why, let me give a simple example:

Router R1 is connected to switch SW1 with a connection to server SRV: R1
<-> SW1 <-> SRV
Router R2 is connected to switch SW2 with a connection to server SRV: R2
<-> SW2 <-> SRV

The server is using R1 as default gateway. Traffic is arriving from the
internet through R2 towards the server. The server will however send
replies back through the default gateway at R1. This is a usual case with
redundant routers - only one will be used as a default gateway but traffic
may come from both.

Initially all will be good. But SW2 is only seeing unidirectional traffic
from R2. No traffic goes from SRV to R2 and thus, after some time, SW2 will
expire the MAC learning for SRV. This has the unfortunate result that SW2
will start flooding traffic to SRV out through all ports.

Then after more time has passed, R2 will renew the ARP binding by sending
out an ARP query to SRV. The server will send back an ARP reply to R2. This
packet from SRV to R2 will pass SW2 and thus have the effect of renewing
the MAC binding at SW2 too. The flooding stops and all is well again. Until
the MAC binding expires and the story repeats.

If the MAC timeout is 5 minutes and the ARP timeout is 20 minutes, which is
very usual, you will have flooding for 15 minutes out of every 20 minutes
interval! Stupid!

Why have vendors not fixed their defaults for this case?

Regards,

Baldur



On Thu, Sep 17, 2020 at 7:51 AM Saku Ytti  wrote:

> On Wed, 16 Sep 2020 at 23:15, Chriztoffer Hansen
>  wrote:
> > On 16/09/2020 04:01, Ryan Hamel wrote:
>
> > > CoPP is always important, and it's not just Mikrotik's with default low
> > > ARP timeouts.
> > >
> > > Linux - 1 minute
> > > Brocade - 10 minutes
> > > Cumulus  - 18 minutes
> > > BSD distros - 20 minutes
> > > Extreme - 20 minutes
> > Juniper - 20 minutes
> > > HP - 25 minutes
> IOS - 4 hours
>
> Why are these considered (by Ryan) low values? Does low have a
> negative connotation here?
>
> ARP timeout should be lower than MAC timeout, and MAC timeout usually
> is 300 seconds. Anything above 300seconds is probably poor BCP for
> default value, as defaults should interoperate in a somewhat sane
> manner.
> Of course operators are free to configure very high ARP timeout, as
> long as they also remember to equally configure higher MAC timeout.
>
> --
>   ++ytti
>

Re: Centurylink having a bad morning?

[Baldur Norddahl]

That is what the 5G router is for...

ons. 2. sep. 2020 19.47 skrev Michael Hallgren :

> While conserving connectivity? 😂
>
>
> --
> *De :* Shawn L via NANOG 
> *Envoyé :* mercredi 2 septembre 2020 13:15
> *À :* nanog
> *Objet :* Re: Centurylink having a bad morning?
>
> We once moved a 3u server 30 miles between data centers this way.  Plug
> redundant psu into a ups and 2 people carried it out and put them in a
> vehicle.
>
>
> Sent from my iPhone
>
> > On Sep 1, 2020, at 11:58 PM, Christopher Morrow 
> wrote:
> >
> > On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert 
> wrote:
> >>
> >>As a coincidence...  I was *thinking* of moving a 90TB SAN (with
> mechanical's) to another rack that way...  skateboard, long fibers and long
> power cords =D
> >>
> >
> > well, what you REALLY need is one of these:
> >  https://www.cru-inc.com/products/wiebetech/hotplug_field_kit_product/
> >
> > and 2-3 UPS... swap to the UPS, then just roll the stack over, plug to
> > utility and done. (minus network transfer)
>

Re: [outages] Major Level3 (CenturyLink) Issues

[Baldur Norddahl]

I believe someone on this list reported that updates were also broken. They
could not add prepending nor modify communities.

Anyway I am not saying it cannot happen because clearly something did
happen. I just don't believe it is a simple case of overload. There has to
be more to it.

ons. 2. sep. 2020 15.36 skrev Saku Ytti :

> On Wed, 2 Sep 2020 at 16:16, Baldur Norddahl 
> wrote:
>
> > I am not buying it. No normal implementation of BGP stays online,
> replying to heart beat and accepting updates from ebgp peers, yet after 5
> hours failed to process withdrawal from customers.
>
> I can imagine writing BGP implementation like this
>
>  a) own queue for keepalives, which i always serve first fully
>  b) own queue for update, which i serve second
>  c) own queue for withdraw, which i serve last
>
> Why I might think this makes sense, is perhaps I just received from
> RR2 prefix I'm pulling from RR1, if I don't handle all my updates
> first, I'm causing outage that should not happen, because I already
> actually received the update telling I don't need to withdraw it.
>
> Is this the right way to do it? Maybe not, but it's easy to imagine
> why it might seem like a good idea.
>
> How well BGP works in common cases and how it works in pathologically
> scaled and busy cases are very different cases.
>
> I know that even in stable states commonly run vendors on commonly run
> hardware can take +2h to finish converging iBGP on initial turn-up.
>
> --
>   ++ytti
>

Re: [outages] Major Level3 (CenturyLink) Issues

[Baldur Norddahl]

I am not buying it. No normal implementation of BGP stays online, replying
to heart beat and accepting updates from ebgp peers, yet after 5 hours
failed to process withdrawal from customers.


ons. 2. sep. 2020 14.11 skrev Saku Ytti :

> On Wed, 2 Sep 2020 at 14:40, Mike Hammett  wrote:
>
> > Sure, but I don't care how busy your router is, it shouldn't take hours
> to withdraw routes.
>
> Quite, discussion is less about how we feel about it and more about
> why it happens and what could be done to it.
>
> --
>   ++ytti
>

Re: Centurylink having a bad morning?

[Baldur Norddahl]

An outage is what it is. I am not worried about outages. We have multiple
transits to deal with that.

It is the keep announcing prefixes after withdrawal from peers and
customers that is the huge problem here. That is killing all the effort and
money I put into having redundancy. It is sabotage of my network after I
cut the ties. I do not want to be a customer at an outlet who has a system
that will do that. Luckily we do not currently have a contract and now they
will have to convince me it is safe for me to make a contract with them. If
that is impossible I guess I won't be getting a contract with them.

But I disagree in that it would be impossible. They need to make a good
report telling exactly what went wrong and how they changed the design, so
something like this can not happen again. The basic design of BGP is such
that this should not happen easily if at all. They did something unwise.
Did they make a route reflector based on a database or something?

Regards,

Baldur

On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho  wrote:

> Exactly. And asking that they somehow prove this won't happen again is
> impossible.
>
> - Mike Bolitho
>
> On Sun, Aug 30, 2020, 8:10 AM Drew Weaver  wrote:
>
>> I’m not defending them but I am sure it isn’t intentional.
>>
>>
>>
>> *From:* NANOG  *On
>> Behalf Of *Baldur Norddahl
>> *Sent:* Sunday, August 30, 2020 9:28 AM
>> *To:* nanog@nanog.org
>> *Subject:* Re: Centurylink having a bad morning?
>>
>>
>>
>> How is that acceptable behaviour? I shall remember never to make a
>> contract with these guys until they can prove that they won't advertise my
>> prefixes after I pull them. Under any circumstances.
>>
>>
>>
>> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins > >:
>>
>> Finally got through on their support line and spoke to level1. The only
>> thing the tech could say was it was an issue with BGP route reflectors and
>> it started about 3am(pacific). They were still trying to isolate the issue.
>> I've tried failing over my circuits and no go, the traffic just dies as L3
>> won't stop advertising my routes.
>>
>>
>>
>> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
>> wrote:
>>
>> Hello,
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity
>> had to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>> As of right now their support portal won’t load:
>> https://www.centurylink.com/business/login/
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>>
>>

Re: Centurylink having a bad morning?

[Baldur Norddahl]

On Sun, Aug 30, 2020 at 5:21 PM Chris Adams  wrote:

> Once upon a time, Baldur Norddahl  said:
> > How is that acceptable behaviour? I shall remember never to make a
> contract
> > with these guys until they can prove that they won't advertise my
> prefixes
> > after I pull them. Under any circumstances.
>
> Umm, then I guess you won't sign a contract with anybody?  I sure
> wouldn't agree to that.  I don't personally write the routing software,
> so I can't guarantee there isn't a bug in there (actually, since it is
> software, I can guarantee there ARE bugs in there).
>
> We'll see if/when they issue an RFO, but software has bugs, and
> configuration errors have entirely unexpected consequences.  It's
> possible some poor design issue was exposed, or it could be some
> basically unforeseeable incident.
>

Not really the point. BGP is designed such that if I take down the link,
the prefixes MUST be withdrawn within reasonable time. The self healing
aspect of the internet entirely depends on this. Clearly they have some
kind of system that does not respect that by design. I am guessing they
have something homebrewn going on with their route reflectors?

It is like a plane. It is impossible to prove or even design a plane that
can never fall out of the sky. But now we had a plane that crashed in a
very bad way, so that plane (Centurylink) is grounded until they can prove
that something like this can not happen again. Which means they need to
redesign whatever the hell they have going on here.

Regards,

Baldur

Re: Centurylink having a bad morning?

[Baldur Norddahl]

How is that acceptable behaviour? I shall remember never to make a contract
with these guys until they can prove that they won't advertise my prefixes
after I pull them. Under any circumstances.

søn. 30. aug. 2020 15.14 skrev Joseph Jenkins :

> Finally got through on their support line and spoke to level1. The only
> thing the tech could say was it was an issue with BGP route reflectors and
> it started about 3am(pacific). They were still trying to isolate the issue.
> I've tried failing over my circuits and no go, the traffic just dies as L3
> won't stop advertising my routes.
>
> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
> wrote:
>
>> Hello,
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity
>> had to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>> As of right now their support portal won’t load:
>> https://www.centurylink.com/business/login/
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>>
>

00:aa:bb:01:23:45

[Baldur Norddahl]

Hello

By accident I noticed several of my VPLS instances have 
00:aa:bb:01:23:45 in the MAC table. We never sent anything just received 
a little traffic from that. Obviously not a real MAC address so I tried 
to search Google for it. I find several hits with apparently ADSL users 
doing pppd (which we do not have).


Anyone have any idea what this could be?

Regards,

Baldur

Re: Bottlenecks and link upgrades

[Baldur Norddahl]

No plan survives contact with the enemy. Your careful made growth
projection was fine until the brass made a deal with some major customer,
which caused a traffic spike. Or any infinite other events that could and
eventually will happen to you.

One hard thing, that almost everyone will get wrong at some point, is
simulating load in the event multiple outages takes some links out, causing
excessive traffic to reroute unto links that previously seemed fine.

Regards,

Baldur


On Sat, Aug 15, 2020 at 10:48 AM Etienne-Victor Depasquale 
wrote:

> I've seen the weekly profiles of traffic sourced from caches for the major
> global services (video, social media, search and general) for a specific
> metro area.
>
> For all services, the weekly profile is a repetition of the daily profile,
> within +/- 20%.
> That is: the weekly profile is obtained from the daily profile within +/-
> 20% of the average daily profile height.
>
> Given this regularity, as suggested by Louie Lee, then it seems that
> growth projections are meaningful.
> That is, the weely profile data, seem to provide a sound empirical basis
> for link upgrades.
>
> Since I'm not an operator, my comments need to be sprinkled with a pinch
> of salt :)
>
> Cheers,
>
> Etienne
>
> On Sat, Aug 15, 2020 at 2:43 AM Louie Lee via NANOG 
> wrote:
>
>> Beyond a pure percentage, you might want to account for the time it takes
>> you stay below a certain threshold. If you want to target a certain link to
>> keep your 95th percentile peaks below 70%, then first get an understanding
>> of your traffic growth and try to project when you will reach that number.
>> You have to decide whether you care about the occasional peak, or the
>> consistent peak, or somewhere in between, like weekday vs weekends, etc.
>> Now you know how much lead time you will have.
>>
>> Then consider how long it will take you to upgrade that link. If it's a
>> matter of adding a couple of crossconnects, then you might just need a
>> week. If you have to ship and install optics, modules, a card, then add
>> another week. If you have to get a sales order signed by senior management,
>> add another week. If you have to put it through legal and finance, add a
>> month. (kidding) If you are doing your annual re-negotiation, well...good
>> luck.
>>
>> It's always good to ask your circuit vendors what the lead times are,
>> then double it and add 5.
>>
>> And sometimes, if you need a low latency connection, traffic utilization
>> levels might not even be something you look at.
>>
>> Louie
>> Peering Coordinator at a start-up ISP
>>
>>
>> On Fri, Aug 14, 2020 at 4:13 PM Radu-Adrian Feurdean <
>> na...@radu-adrian.feurdean.net> wrote:
>>
>>> On Wed, Aug 12, 2020, at 09:31, Hank Nussbacher wrote:
>>> > At what point do commercial ISPs upgrade links in their backbone as
>>> > well as peering and transit links that are congested?  At 80%
>>> capacity?
>>> >  90%?  95%?
>>>
>>> Some reflections about link capacity:
>>> At 90% and over, you should panic.
>>> Between 80% and 90% you should be (very) scared.
>>> Between 70% and 80% you should be worried.
>>> Between 60% and 70% you should  seriously consider speeding up the
>>> upgrades that you effectively started at 50%, and started planning since
>>> 40%.
>>>
>>> Of course, that differs from one ISP to another. Some only upgrade after
>>> several months with at least 4 hours a day, every day (or almost) at over
>>> 95%. Others deploy 10x expected capacity, and upgrade well before 40%.
>>>
>>
>
> --
> Ing. Etienne-Victor Depasquale
> Assistant Lecturer
> Department of Communications & Computer Engineering
> Faculty of Information & Communication Technology
> University of Malta
> Web. https://www.um.edu.mt/profile/etiennedepasquale
>

Re: Bottlenecks and link upgrades

[Baldur Norddahl]

I expect my hardware does not have such a metric, but maybe it should have.
Max queue length tell us how full the link is with respect to microbursts.


tor. 13. aug. 2020 15.28 skrev Mike Hammett :

> I suppose it would depend on if your hardware has an OID for what you want
> to monitor.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> --
> *From: *"Baldur Norddahl" 
> *To: *nanog@nanog.org
> *Sent: *Thursday, August 13, 2020 8:20:26 AM
> *Subject: *Re: Bottlenecks and link upgrades
>
> Is it possible to do and is anyone monitoring metrics such as max queue
> length in 5 minutes intervals? Might be a better metric than average load
> in 5 minutes intervals.
>
> Regards
>
> Baldur
>
>

Re: Bottlenecks and link upgrades

[Baldur Norddahl]

Is it possible to do and is anyone monitoring metrics such as max queue
length in 5 minutes intervals? Might be a better metric than average load
in 5 minutes intervals.

Regards

Baldur

Re: Is there *currently* a shortage of IPv4 addresses?

[Baldur Norddahl]

IP address space is no longer free. But an ISP or hosting company is a
trader of addresses now and like everything else we do, there is an
opportunity to make a margin.

Say the provider bought at $12 per address and assuming IPv4 is needed for
at least 10 years, that would only be .1 USD/month.

But does that mean it is unfair to claim a $2 rent on that? What if the
service has other components that are equally cheaper?

Regards
Baldur


tir. 4. aug. 2020 21.34 skrev Anne P. Mitchell, Esq. :

> I know that a shortage of IPv4 addresses has been anticipated for quite
> some time (literally decades), however, is there a shortage *right now*?
>
> I ask, because Liquid Web is using it as an excuse to raise their prices:
>
> "We're contacting you today to inform you of a change to your account. As
> you may know, the global shortage of IPv4 addresses (
> https://www.ripe.net/manage-ips-and-asns/ipv4/ipv4-run-out) continues to
> impact web hosting companies around the world. ... Effective August 31st,
> we will be updating our per IPv4 address price to $2.00 per IP."
>
> Anne
>
> --
> Anne P. Mitchell,  Attorney at Law
> Dean of Cyberlaw & Cybersecurity, Lincoln Law School
> CEO, SuretyMail Email Reputation Certification
> Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam law)
> Board of Directors, Denver Internet Exchange
> Chair Emeritus, Asilomar Microcomputer Workshop
> Former Counsel: Mail Abuse Prevention System (MAPS)
>
>

Re: BGP route hijack by AS10990

[Baldur Norddahl]

On Mon, Aug 3, 2020 at 3:54 PM Job Snijders  wrote:

> On Mon, Aug 03, 2020 at 02:36:25PM +0200, Alex Band wrote:
> > According to the information I received from the community[1], you
> > should read PR1461602 and PR1309944 before deploying.
> >
> > [1] https://rpki.readthedocs.io/en/latest/rpki/router-support.html
>
> My take on PR1461602 is that it can be ignored, as it appears to only
> manifest itself in a mostly cosmetic way: initial RTR session
> establishment takes multiple minutes, but once RTR sessions are up
> things work smoothly.
>
> Under no circumstances should you enable RPKI ROV functionality on boxes
> that suffer from PR1309944. That one is a real showstopper.
>
>
We suffered a series of crashes that led to JTAC recommending disabling
RPKI. We had a core dump which matches PR1332626 which is confidential, so
I have no idea what it is about. Apparently what happened was the server
running the RPKI validation server rebooted and the service was not
configured to automatically restart. Also we did not have it redundant nor
did we monitor the service. So we had no working RPKI validation server and
that apparently caused the MX204 to become unstable in various ways. It
might run for a day but it would do all sorts of things like packet loss,
delays and generally be "strange". The first crash caused BGP, ssh and
subscriber management to be down, but LDP, OSPF, SNMP to be up. It became a
black hole we could not login to.  The worst possible kind of crash for a
router. We had to go onsite and pull the power.

The router appears to run fine after disabling RPKI. I suppose starting the
validation service may also fix the issue. But I am not going to go there
until I know what is in that PR and also I feel the RPKI funktion needs to
be failsafe before we can use it. I know we are at fault for not deploying
the validation service in a redundant setup and for failing at monitoring
the service. But we did so because we thought it not to be too important,
because a failed validation service should simply lead to no validation,
not a crashed router.

This is on JUNOS 20.1R1.11.

Regards,

Baldur

Re: BGP route hijack by AS10990

[Baldur Norddahl]

How do you know that none of the prefixes had ROA? The ones that had got
stopped by Telias filter, so we would never know.

This is exactly the situation where RPKI already works. My and yours
prefixes, provided you like me have ROAs, will not be leaked through Telia
and a number of other large transits. Even if they did not have proper
filters in place.

Driving without RPKI / ROA is like driving without a seatbelt. You are fine
until the day someone makes a mistake and then you wish you did your job at
signing those prefixes sooner.

Regards,

Baldur


On Fri, Jul 31, 2020 at 3:35 PM Mark Tinka  wrote:

>
>
> On 31/Jul/20 03:57, Aftab Siddiqui wrote:
> > Not a single prefix was signed, what I saw. May be good reason for
> > Rogers, Charter, TWC etc to do that now. It would have stopped the
> > propagation at Telia.
>
> While I am a huge proponent for ROA's and ROV, it is a massive
> expectation to req filtering to work on the basis of all BGP
> participants creating their ROA's. It's what I would like, but there is
> always going to be a lag on this one.
>
> If none of the prefixes had a ROA, no amount of Telia's shiny new "we
> drop invalids" machine would have helped, as we saw with this incident.
> ROV really only comes into its own when the majority of the Internet has
> correct ROA's setup. In the absence of that, it's a powerful but
> toothless feature.
>
> So while I will continue pushing for the rest of the world to create
> ROA's, turn on RPKI and enable ROV, I'll also advocate that operators
> continue to have both AS- and prefix-based filters. Not either/or, but
> both. Also, max-prefix as a matter of course.
>
> Mark.
>