Re: Authoritative Resources for Public DNS Pinging

[Mark Tinka]

On 2/11/22 15:33, Tom Beecher wrote:


I respectfully strongly disagree on 'need'.

Let's perform a thought experiment. Assert that 8.8.8.8 was expressly 
codified by Google to be a designated ICMP endpoint, and that for 100% 
of ICMP echo requests they receive, they guarantee an echo-reply will 
be sent. There are countless reasons , even with that (unreasonable) 
assumption of 100% uptime of the endpoint, that echo-requests may not 
reach them, or that echo-replies may be sent but not reach the 
originating source. Extend this idea even further. Assert that it is 
now not just Google running it, and the largest networks in the world 
all agree to anycast it from their networks.Assert is still guaranteed 
to respond to 100% if all echo requests it receives, wherever it 
receives. ( An even more unreasonable assertion than before!) There 
are STILL countless reasons why an endpoint may, at times, have that 
simple ICMP check fail.


The prediciate assumption that "pinging one destination is a valid 
check that my internet works' is INCORRECT. There is no magical 
unicorn that could be built that could make that true, and 'they're 
gonna do it anyways' is a poor excuse to even consider it.


This is a mistake many of us have made. I'll openly admit I made it 20 
years ago. Like someone on the outages list I think mentioned, I had 
built a couple SLA checks that triggered some routing changes to occur 
based on their status, and I thought I was super hot shit. Until I had 
to drive an hour through a blizzard to bring my routers back up after 
my incorrect assumptions knocked my entire company (an ISP) offline. 
Sometimes these are lessons people need to learn, but it's also 
helpful to point out to others why what they are trying to do is a bad 
idea so they can( if they chose to) learn from our prior mistakes.


I said "liveliness detection", not that "is the Internet working?".

The most basic thing is to check that the link between your device and 
your ISP is working, and a simple ping (of 8.8.8.8, nowadays) is 
typical, either by hand, or by your CPE. It does not guarantee that the 
rest of the Internet is alive, but it does tell you that if there is a 
problem further upstream, it's not yours, but likely either your ISP's, 
or the remote network you are trying to reach.


There is a need for that. It just so happens that 8.8.8.8 fills that 
need, at present. If 8.8.8.8 goes away, folk will find something else to 
use for first-line liveliness detection.


Mark.

Re: Authoritative Resources for Public DNS Pinging

[Mark Tinka]

On 2/11/22 16:58, Jon Lewis wrote:



I have to admit, I haven't read most of this thread, but I am well 
aware of the issues with both end users and "routers" / firewalls 
pinging 8.8.8.8 as a means of verifying "that path to the Internet is 
working".  I know GOOG doesn't appreciate the amount of ICMP echo 
requests their 8.8.8.8 instances receive, and that at various 
times/places, that ICMP traffic is/has been policed by GOOG.


So...here's a pair of "what if"s:

What if instead of pinging 8.8.8.8, all these things using it to "test 
the Internet" sent it DNS requests instead?  i.e.

GOOG=$(dig +short @8.8.8.8 google.com)
if [ -z "$GOOG" ] ; then
  echo FAIL
fi Would that make things better or worse for GOOG (Trading lots more 
DNS requests for the ICMP echo requests)?


Could work for devices, but more difficult for Jane.




8.8.8.8 is already anycasted.  What if each large ISP (for whatever 
definition of large floats your boat) setup their own internal 
instance(s) of 8.8.8.8 with a caching DNS server listening, and 
handled the traffic without bothering GOOG?  For users using 8.8.8.8 
as a lighthouse, this would change the meaning of their test...i.e. a 
response means their connection to their ISP is up, and the ISP's 
network works at least enough to reach an internal 8.8.8.8, but the 
question of their connectivity to the rest of the Internet would be 
unanswered.


Something tells me Google (or Cloudflare, or Quad9, or e.t.c.) would not 
consider that a good thing, for them :-).


Mark.

Re: Authoritative Resources for Public DNS Pinging

[Mark Tinka]

On 2/11/22 22:43, Mike Lewinski via NANOG wrote:


On a related note, I just discovered a NID that has 1.1.1.1 assigned to the outband 
interface by default, and it is apparently not user modifiable. So, not only can these 
devices never use 1.1.1.1 for name resolution, but attempts to determine "is the 
circuit up" by pinging it will always return bad information. To really pour salt on 
the wound, this device has no physical outbound interface (likely why the UI doesn't 
allow changing it).

Bug report filed. I don't really want to use it for either purpose, but 
hopefully a fix saves someone else the headache. And it really chaps my  
when public addresses get used this way.


Do you know if this was codified prior to 1.1.1.1 being taken over by 
Cloudflare?


Mark.

Re: junos config commit question

[Mark Tinka]

On 2/12/22 00:54, Jon Lewis wrote:



Also, get into the habit of never doing a commit without first doing
top show | compare
so you can see what your change is actually doing to the whole config. 
i.e. if you did a show | compare at the top of the config and saw the 
entire interfaces section of the config was "removed" in the resulting 
config diff, you probably wouldn't commit.


That is always my habit, with plenty of muscle memory... "show | compare".

I have often found it interesting how many folk have muscle memory for 
"commit and-quit", including Juniper's own staff when I've had the 
pleasure of being with them on a PoC. It's almost as if I missed an 
entire period of Junos where that was deemed to be good practice :-).


Mark.

Re: Authoritative Resources for Public DNS Pinging

[Mark Tinka]

On 2/11/22 14:27, Mike Hammett wrote:

The device that caused this whole conversation has failover 
functionality. Both interfaces ping an FQDN (that resolves to 8.8.8.8 
and 1.1.1.1, with the device only latching on to one of those). If any 
of those meet the failure threshold, that interface is taken out of 
the traffic flow.



So because someone else built a device (without a meaningful 
configuration to set otherwise), 8.8.8.8 went down for ICMP, and thus 
Internet ports began flapping, despite the Internet as a whole working 
just fine.


Pretty amazing, isn't it?

Mark.

Re: junos config commit question

[Dale Shaw]

Hey Mark,

On Sat, 12 Feb 2022 at 8:25 pm, Mark Tinka  wrote:

>
> I have often found it interesting how many folk have muscle memory for
> "commit and-quit", including Juniper's own staff when I've had the
> pleasure of being with them on a PoC. It's almost as if I missed an
> entire period of Junos where that was deemed to be good practice :-).


That’s definitely a practice guaranteed to result in needing to drive to
the DC.

I wonder if it creeps into some folks’ MO because the control plane on many
platforms is soo slow to commit. Many don’t know that a “commit check”
is sufficient to confirm a commit, but even that can take a long time.

Cheers,
Dale

RE: Authoritative Resources for Public DNS Pinging

[Mike Lewinski via NANOG]

> Do you know if this was codified prior to 1.1.1.1 being taken over by 
> Cloudflare?

Yes, I'm sure it was.

Re: Authoritative Resources for Public DNS Pinging

[Mark Tinka]

On 2/12/22 16:43, Mike Lewinski via NANOG wrote:


Yes, I'm sure it was.


Then probably rhymes with the days of "admin/admin".

If they have been pushing out security and OS updates since then, and 
still keep 1.1.1.1 coded, that is purely their fault.


Mark.

Re: VPN recommendations?

[Christian de Larrinaga via NANOG]

Intriguing. This week I started to look around for new wireguard 
implementation tools and appliances. I've used openvpn and ipsec 
in the main although last month put together a 10x and IPv6 
wireguard net in my home and out to two vps hosts which is 
handy. For my own use this is ok -ish, but I am not so sure about 
keeping track of the configs, managing users and adding configs as 
a network grows. In other words I want help when scaling wg and 
handling change particularly if I am managing nets for other 
projects or delegating. 

Tailscale, ZeroTier and some others are doing a great job I feel 
and no doubt have a handle on that. I've not tried them as yet. 

Because I do like to have options that are not mediated I have 
kept looking as much for my own curiousity and education as for 
deploying a service in anger. But having a toolset that can 
support the latter capability has to be the aim to work towards.


I've found a few potentially interesting more recent projects and 
am intending to start to test deploy some of these in sequence to 
see how I get on. I think I'll start wth
https://github.com/gravitl/netmaker Please note I've only reviewed 
the documentation. I've not yet played with it.  

This seems to  offer at an early stage in its development a 
webappliance (optionally) with CoreDNS if you want  naming support 
and IPv6 and at least some client management features. It claims 
to be fast but that can be tested. It also is deployable as a 
docker/kubernetes k8 which is intriguing when deploying and 
managing containers between multiple hosts across data centres. 
It uses a mongodb licence which may or may not be a problem.


If one plays with IPSEC then I guess one could run wg through 
IPSEC but is there any point unless you already have an IPSEC 
branch and don't want to take it down whilst adding wg for a new 
class of devices/userbase?   

I'd be interested in sharing experiences and advice (offlist) and 
delighted to learn from  wireguard and vpn's clueful folk. 

thank you for an interesting discussion. 



Christian

William Herrin  writes:

On Fri, Feb 11, 2022 at 10:35 AM Dan Sneddon  
wrote:
1) IPSEC does not lend itself to dynamic routing or dynamic 
configuration. It is very much a static set-it-and-forget-it 
technology, but that doesn’t work in a dynamically changing 
environment.


Hi Dan,

Depending on how you configure it, IPSEC can work fine with 
dynamic

routing. The thing to understand is that IPSec has two modes:
transport and tunnel. Transport is between exactly two IP 
addresses
while tunnel expects a broader network to exist on at least one 
end.
"Tunnel" mode is what everyone actually uses but you can 
deconstruct
it: it's built up from transport mode + a tunnel protocol (gre 
or ipip

I don't remember which) + implicit routing and firewalling which
wreaks havoc on dynamic routing. Now, it turns out that you can
instead configure IPSec in transport mode, configure the tunnel
separately and leave out the implicit firewalling.

This may not apply to William Herrin’s (OP) use case of a VPN 
appliance


It's not relevant to my situation, no. I need the VPN to 
establish a
statically addressed clean layer 3 on top of dynamically 
addressed and
natted endpoints to support the next appliance in the chain 
where
dynamic addressing is not possible. I don't actually care if it 
adds
security; it just needs to establish that statically addressed 
layer.
Oh yeah, and it has to be listed under "virtual private network" 
on

the government NIAP list.
https://www.niap-ccevs.org/product/PCL.cfm?ID624=34

Regards,
Bill Herrin



--
Christian de Larrinaga 
https://firsthand.net

Re: (Free)RADIUS Front-End

[Mark Tinka]

For posterity, finally went with Splynx.

Really awesome product, covering not only RADIUS but also CRM, billing, 
invoicing, remote integration, e.t.c.


Just in case anyone else ends up having the same requirement.

Mark.

On 9/20/21 09:19, Mark Tinka wrote:



On 9/20/21 02:16, Philip Loenneker wrote:
Splynx is a commercial product designed to be an entire package for 
running an ISP, including billing etc. It uses FreeRadius in the 
backend which chains into their own RADIUS system. Integration for 
MikroTik routers is very extensive, but we have had it working with a 
variety of other BNGs too including Cisco and Linux-based systems. 
You can add in RADIUS dictionaries and customise the router profiles 
via the GUI to send whatever VSAs you require, including for COA. The 
APIs on it are extensive, making automation of service provisioning 
relatively easy. The IPAM in it is fairly basic, primarily ensuring 
that you don't re-use IPs between multiple services. IPv6 support is 
included, but primarily for IPv6-PD rather than interface IPs, 
however I have managed to get a BNG to assign an IPv4 address, an 
IPv6 interface address and IPv6-PD all from the Splynx profile. The 
accounting is ok, allowing you to apply bandwidth caps on users if 
required, including different speeds for different times of day.


www.splynx.com


Many thanks, Philip. Looks awesome!

Mark.

Re: Fiber contractor in Washington state

[Matthew Barker]

Late reply, but I had https://www.zerodbcomm.com do a decent sized splice 
project out in Liberty Lake for us and they did greatwork.


Not sure if they do dirt work, but they might have a local recommendation 
or partner for trenching or bore.


-Matt


On Wed, 9 Feb 2022, Aaron C. de Bruyn via NANOG wrote:


Cascade Networks out of Longview Washington does (or used to do) fiber 
installs.They got bought out by Wave a few years ago, but
I think their fiber division is still active.

https://cni.net/

-A

On Tue, Feb 8, 2022 at 4:50 PM Ross Tajvar  wrote:
  Hi all,
I'm looking for a fiber contractor to trench some fiber on private property and 
then splice it inside. The work will be in
Washington state, north of Spokane. Does anyone have recommendations? On- and 
off-list welcome.

Thanks,
Ross

Re: DMARC ViolationDKIM ViolationRe: junos config commit question

[Paschal Masha]

More like driving with the hand break still engaged. 

Always, after changing the candidate config, run " show | compare" - loving 
junos. 

Regards 
Paschal Masha | Engineering 
Skype ID: paschal.masha 


From: "Mark Tinka"  
To: "nanog"  
Sent: Saturday, February 12, 2022 12:23:09 PM 
Subject: DMARC ViolationDKIM ViolationRe: junos config commit question 

On 2/12/22 00:54, Jon Lewis wrote: 

> 
> Also, get into the habit of never doing a commit without first doing 
> top show | compare 
> so you can see what your change is actually doing to the whole config. 
> i.e. if you did a show | compare at the top of the config and saw the 
> entire interfaces section of the config was "removed" in the resulting 
> config diff, you probably wouldn't commit. 

That is always my habit, with plenty of muscle memory... "show | compare". 

I have often found it interesting how many folk have muscle memory for 
"commit and-quit", including Juniper's own staff when I've had the 
pleasure of being with them on a PoC. It's almost as if I missed an 
entire period of Junos where that was deemed to be good practice :-). 

Mark. 

Re: junos config commit question

[Paschal Masha]

Not long enough to have drive to the DC in the middle of the night :) 

Even "commit confirmed x" is a shield, a better one. 


Regards 
Paschal Masha | Engineering 
Skype ID: paschal.masha 


From: "Dale Shaw"  
To: "Mark Tinka"  
Cc: "nanog"  
Sent: Saturday, February 12, 2022 12:39:28 PM 
Subject: Re: junos config commit question 

Hey Mark, 

On Sat, 12 Feb 2022 at 8:25 pm, Mark Tinka  wrote: 



I have often found it interesting how many folk have muscle memory for 
"commit and-quit", including Juniper's own staff when I've had the 
pleasure of being with them on a PoC. It's almost as if I missed an 
entire period of Junos where that was deemed to be good practice :-). 



That’s definitely a practice guaranteed to result in needing to drive to the 
DC. 

I wonder if it creeps into some folks’ MO because the control plane on many 
platforms is soo slow to commit. Many don’t know that a “commit check” is 
sufficient to confirm a commit, but even that can take a long time. 

Cheers, 
Dale 

Re: (Free)RADIUS Front-End

[Gaurav Kansal]

I have also developed Free Radius based AAA (Authentication , Authorisation and 
Accounting) solution , and we have replaced Cisco ISE with our in-house 
developed product. 
More than 30K clients are getting authenticated and managed through this portal.

In case, if anyone needs any help or suggestions in FR, then feel free to 
connect.

Thanks,
Gaurav Kansal


> On 12-Feb-2022, at 20:45, mark@tinka.africa wrote:
> 
> For posterity, finally went with Splynx. 
> 
> Really awesome product, covering not only RADIUS but also CRM, billing, 
> invoicing, remote integration, e.t.c.
> 
> Just in case anyone else ends up having the same requirement.
> 
> Mark.
> 
> On 9/20/21 09:19, Mark Tinka wrote:
>> 
>> 
>> On 9/20/21 02:16, Philip Loenneker wrote: 
>>> Splynx is a commercial product designed to be an entire package for running 
>>> an ISP, including billing etc. It uses FreeRadius in the backend which 
>>> chains into their own RADIUS system. Integration for MikroTik routers is 
>>> very extensive, but we have had it working with a variety of other BNGs too 
>>> including Cisco and Linux-based systems. You can add in RADIUS dictionaries 
>>> and customise the router profiles via the GUI to send whatever VSAs you 
>>> require, including for COA. The APIs on it are extensive, making automation 
>>> of service provisioning relatively easy. The IPAM in it is fairly basic, 
>>> primarily ensuring that you don't re-use IPs between multiple services. 
>>> IPv6 support is included, but primarily for IPv6-PD rather than interface 
>>> IPs, however I have managed to get a BNG to assign an IPv4 address, an IPv6 
>>> interface address and IPv6-PD all from the Splynx profile. The accounting 
>>> is ok, allowing you to apply bandwidth caps on users if required, including 
>>> different speeds for different times of day. 
>>> 
>>> www.splynx.com  
>> 
>> Many thanks, Philip. Looks awesome! 
>> 
>> Mark. 
>> 
> 

Re: VPN recommendations?

[Grant Taylor via NANOG]

On 2/11/22 12:35 PM, William Herrin wrote:
The thing to understand is that IPSec has two modes: transport and 
tunnel. Transport is between exactly two IP addresses while tunnel 
expects a broader network to exist on at least one end.


That is (syntactically) correct.  However, it is possible to NAT many 
LAN IPs (say RFC 1918) to one single Internet IP (say from a SOHO ISP) 
and use IPSec /Transport/ Mode to a single remote IP.  The IPSec sees 
exactly two IPs.



"Tunnel" mode is what everyone actually uses


I may be enough of an outlier that I'm a statistical anomaly.  But I'm 
using IPSec /Transport/ Mode between my home router and my VPSs.  I have 
a tiny full mesh of IPSec /Transport/ Mode connections.


Using the aforementioned many-to-one NAT, my home LAN systems access the 
single globally routed IP of each of my VPSs without any problem.


Aside:  I did have to tweak MTU for LAN traffic going out to the VPS IPs.

So -1 for '"Tunnel" mode is what everyone actually uses', and +1 for 
/Transport/ Mode


but you can deconstruct it: it's built up from transport mode + 
a tunnel protocol (gre or ipip I don't remember which) + implicit 
routing and firewalling which wreaks havoc on dynamic routing.


I question the veracity of that statement.  It may be that's what many 
implementations / administration systems do.  But I really thought that 
IPSec /Tunnel/ Mode was more than just IPSec /Transport/ Mode combined 
with some tunneling protocol.


Now, it turns out that you can instead configure IPSec in transport 
mode, configure the tunnel separately and leave out the implicit 
firewalling.


Agreed.  I feel like this speaks to implementation / management systems 
that are built on top of IPSec.


It's not relevant to my situation, no. I need the VPN to establish 
a statically addressed clean layer 3 on top of dynamically addressed 
and natted endpoints to support the next appliance in the chain where 
dynamic addressing is not possible. I don't actually care if it adds 
security; it just needs to establish that statically addressed layer.


It sounds to me like you don't even actually need encryption of a 
typical VPN and might be able to use something like GRE+key or IPSec 
/Tunnel/ Mode with AH without ESP.


Oh yeah, and it has to be listed under "virtual private network" 
on the government NIAP list.

https://www.niap-ccevs.org/product/PCL.cfm?ID624=34


Oh joy.  Layer 8 - politics



--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: VPN recommendations?

[Nathan Angelacos]

On Sat, 2022-02-12 at 13:24 -0700, Grant Taylor via NANOG wrote:
> On 2/11/22 12:35 PM, William Herrin wrote:
> > The thing to understand is that IPSec has two modes: transport and 
> > tunnel. Transport is between exactly two IP addresses while tunnel 
> > expects a broader network to exist on at least one end.
> 
> That is (syntactically) correct.  However, it is possible to NAT many
> LAN IPs (say RFC 1918) to one single Internet IP (say from a SOHO
> ISP) 
> and use IPSec /Transport/ Mode to a single remote IP.  The IPSec sees
> exactly two IPs.
> 
> > "Tunnel" mode is what everyone actually uses
> 
> I may be enough of an outlier that I'm a statistical anomaly.  But
> I'm using IPSec /Transport/ Mode between my home router and my VPSs. 
> I have a tiny full mesh of IPSec /Transport/ Mode connections.
> 

+1 on *cough* enterprise networks.

> Using the aforementioned many-to-one NAT, my home LAN systems access
> the single globally routed IP of each of my VPSs without any problem.
> 

+1

> Aside:  I did have to tweak MTU for LAN traffic going out to the VPS
> IPs.

+1

> 
> So -1 for '"Tunnel" mode is what everyone actually uses', and +1 for 
> /Transport/ Mode 

+1

Re: junos config commit question

[Nick Suan via NANOG]

You're correct. 

This the lab setup and rstp was set to the default, so I only got the commit 
check to pass only when I deleted [protocols rstp].


On Fri, Feb 11, 2022, at 8:09 PM, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote:
> Nick Suan via NANOG writes:
>> I was actually interested to see if the EX series would let me do this, and i
>> t turns out that if STP is enabled on any of the switch interfaces, it won't:
>> tevruden@core-02# commit check 
>> [edit protocols rstp]
>>   'interface'
>> XSTP : Interface ge-0/0/0.0 is not enabled for Ethernet Switching
>> error: configuration check-out failed
>
> Do you have any rstp-specific overrides in your config? E.g. we
> have things like this in some of ours:
>
>   rstp {
>   interface ge-0/0/45 {
>   cost 1000;
>   mode point-to-point;
>   }
>   interface ge-1/0/45 {
>   cost 1000;
>   mode point-to-point;
>   }
>   interface ae4;
>   bpdu-block-on-edge;
>   }
>
> With the interfaces gone I would expect the commit check to fail.
>
> --lyndon

Re: VPN recommendations?

[William Herrin]

On Sat, Feb 12, 2022 at 12:26 PM Grant Taylor via NANOG  wrote:
> On 2/11/22 12:35 PM, William Herrin wrote:
> > The thing to understand is that IPSec has two modes: transport and
> > but you can deconstruct it: it's built up from transport mode +
> > a tunnel protocol (gre or ipip I don't remember which) + implicit
> > routing and firewalling which wreaks havoc on dynamic routing.
>
> I question the veracity of that statement.  It may be that's what many
> implementations / administration systems do.  But I really thought that
> IPSec /Tunnel/ Mode was more than just IPSec /Transport/ Mode combined
> with some tunneling protocol.

It's tunnel mode plus a tunneling protocol plus some implicit routing
and firewalling which gets in the way of dynamic routing.

Try it if you don't believe me. Set up tunnel mode ipsec manually on
two nodes (no IKE) and get them talking to each other. Then change one
to transport mode and add I think it's an IPIP tunnel but I don't
remember for certain. And add the appropriate routes into the tunnel
virtual device. You'll find they talk.

What did you think IPSec was doing? Transport mode encrypts the layer
4 and up of the packet between two machines; it doesn't encapsulate
it. When they added tunnel mode, the inner layer 3 had to go
somewhere.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/