Re: NTP Sync Issue Across Tata (Europe)

[Forrest Christian (List Account)]

The recommendation tends to be the following:

1) Run your GPS-derived NTP appliances, but DO NOT point end-user clients
at it.
2) Run a set of internal NTPd servers, and configure them to pull time from
all of your GPS-derived NTP servers, AND trusted public NTP servers
3) Point your clients at the internal NTPd servers.

Note that it's not entirely unreasonable to go out and buy numerous GPS
appliances, deploy them at multiple locations, and point your NTPd servers
at those.   With enough sites, your NTPd server will skip over any
defective NTP appliance.At some point, using publicly available NTP
sources is redundant unless one wants to mitigate away the risks behind
failure of the GPS system itself.

What I'm advocating against is the seemingly common practice to go buy an
off-the-shelf lower-cost GPS-NTP appliance (under $1K or so), stick an
antenna in a window or maybe on the rooftop, and point all your devices at
that device.This is asking for a failure for reasons I've covered
previously.   Robust time needs multiple time sources in order to mitigate
against any one source being spoofed or jammed.   The more time sources you
incorporate into your solution, the less likely it is that any one of them
going haywire would cause a timing failure.

On Wed, Aug 9, 2023 at 6:12 PM Mel Beckman  wrote:

> Seth,
>
> My point exactly. Use GPS as primary, and have anti-PS back up, and if you
> want automatic fail over, do that in an intermediate server on your site
> that makes a conscious test and decision to fail over to regular NTP
>
> -mel via cell
>
> > On Aug 9, 2023, at 5:01 PM, Seth Mattinen via NANOG 
> wrote:
> >
> > On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
> >> Note that NIST operates a pool of 24 time servers for public use.
> These are spread across four different locations in two different states.
> My understanding is that they all get their time directly from the official
> NIST clocks without GPS or NTP being involved.
> >
> > I used to jump through all the hoops for that but honestly I like the
> appliances better (they are also PTP grandmaster clocks). I can always
> disable the GPS inputs if any of the doom and gloom actually comes to pass.
> >
> > ~Seth
>


-- 
- Forrest

Re: NTP Sync Issue Across Tata (Europe)

[Forrest Christian (List Account)]

Let me address your points:

First, the spoofing does mess with the timing stream.  To not mess with the
timing stream, the entity doing the spoofing would have to have
high-quality NTP-synchronized clocks and somehow generate the GPS I-Q data
such that it was perfectly synchronized with real-time.   Quite frankly,
doing this is orders of magnitude harder than spoofing time and position,
especially when operating in a clandestine manner.   Note that position is
derived from time, not the other way around.

As far as the relationship to your antenna goes:   while it is true that a
high-quality GNSS antenna with a pattern pointing up will reduce the
likelihood of ground-based interference messing with your signal, the
rejection rarely exceeds 10dB.  This means that any signal 10dB louder than
the GPS signals will override your GPS signals.  This level of signal is
trivial to produce.

Let's assume you have a typical GPS-derived NTP server using a typical
commercially available timing GNSS module.  To convince that receiver that
it was a different time, I'd need to have an SDR that would operate in the
GPS band.  These are widely available for under $500.  You'd also need a
laptop and a download of a GPS simulator from GitLab.   With a total
investment of $500 (assuming I already have a laptop), I now have a system
that can generate a GPS signal to convince your GPS receiver that it's any
time at all.  If you're a tech neophyte, there are youtube videos on how to
do this.

All I need to do now is add appropriate antennas and/or amplifiers to
overcome the official GNSS signals.   As you pointed out, depending on the
location and directivity of your antenna, this is either trivial or becomes
slightly more difficult.   If I can see your antenna, it becomes a lot
cheaper as I just need a relatively low-powered amplifier and a highly
directional antenna.   If I can't see your antenna, I would opt for a
higher-power amplifier and a less directional transmit antenna to blanket a
wide area with the spoofed signal.

The paragraph above assumes I'm trying to convince your NTP server it's a
different time.   If I just want to deny you time, it gets cheaper and
easier.   All I need is a 1.2 GHz oscillator coupled to an antenna.  There
are units like this available for under $10, delivered.  These block GPS
trackers on trucks and/or private automobiles.   Build your own and you can
get a watt or two to shove into a tiny antenna for not a lot more.
Guaranteed to Jam anything within a couple of blocks.

I wish I could say this is uncommon to happen, but I have seen it time and
time again with customers (I design and sell GNSS Receivers used for
precision timing on various WISP access points).   It doesn't necessarily
need to be intentional - it's not uncommon for a radio transmitter to fail
so that it puts a spur out on 1.2 GHz and/or other GNSS bands.Two
particular recent events happened in January and October of last year.  In
the first, a transmitter started emitting noise on the L1 band near DIA
airport, wiping out GPS on the ground for a 50-mile radius and 230 miles in
the air.   It took 33 hours to find and resolve this particular issue.  See
https://www.cisa.gov/sites/default/files/publications/CISA-Insights_GPS-Interference_508.pdf
.   In the second, similar event, around DFW, an as-yet unidentified noise
source took out GPS for 24 hours - although it took another 20 hours for
everything to recover.

There was a recent event at a site in California that I was peripherally
involved with (as the vendor of an affected GPS device) where a
commercially available microwave point-to-point link started emitting
signals in the GPS band and took out GPS for an extended period at the
site.  In that case, every single GPS receiver at the site could not
receive GPS signals until the errant radio was determined and shut down.
 It took a while as it required a lot of "turn off equipment one by one
until the signal goes away" troubleshooting.

I agree that GPS timing vendors are continuously striving to improve the
reliability and interference robustness of their GPS systems.   There are
definitely really cool solutions on the market today, for example Microchip
Technologies's bluesky GPS firewall which develops it's own internal clock,
and then uses that to deliver precision time downstream by transmitting an
internally-developed GPS signal (just like a spoofer would).   But most
people who develop their GPS time don't select vendors with this level of
robustness.  Instead they almost always use a commerical off the shelf GPS
module which lack many of the features you describe.

A good reference on how to harden GPS systems is at
https://www.cisa.gov/sites/default/files/2023-02/Improving_the_Operation_and_Development_of_Global_Positioning_System_%28GPS%29_Equipment_Used_by_Critical_Infrastructure_S508C.pdf
.


On Wed, Aug 9, 2023, 12:52 PM Mel Beckman  wrote:

> While GPS spoofing is technically possible, all the extant spoofi

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Seth,

My point exactly. Use GPS as primary, and have anti-PS back up, and if you want 
automatic fail over, do that in an intermediate server on your site that makes 
a conscious test and decision to fail over to regular NTP

-mel via cell

> On Aug 9, 2023, at 5:01 PM, Seth Mattinen via NANOG  wrote:
> 
> On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
>> Note that NIST operates a pool of 24 time servers for public use.These 
>> are spread across four different locations in two different states.  My 
>> understanding is that they all get their time directly from the official 
>> NIST clocks without GPS or NTP being involved.
> 
> I used to jump through all the hoops for that but honestly I like the 
> appliances better (they are also PTP grandmaster clocks). I can always 
> disable the GPS inputs if any of the doom and gloom actually comes to pass.
> 
> ~Seth

Re: NTP Sync Issue Across Tata (Europe)

[Seth Mattinen via NANOG]

On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
Note that NIST operates a pool of 24 time servers for public use.  
  These are spread across four different locations in two different 
states.  My understanding is that they all get their time directly from 
the official NIST clocks without GPS or NTP being involved.




I used to jump through all the hoops for that but honestly I like the 
appliances better (they are also PTP grandmaster clocks). I can always 
disable the GPS inputs if any of the doom and gloom actually comes to pass.


~Seth

Re: NTP Sync Issue Across Tata (Europe)

[Forrest Christian (List Account)]

Note that NIST operates a pool of 24 time servers for public use.   These
are spread across four different locations in two different states.  My
understanding is that they all get their time directly from the official
NIST clocks without GPS or NTP being involved.

You can also request a symmetrical key,  exchanged via paper mail,  for
four of them if you would like to run ntp encryption.

See https://tf.nist.gov/tf-cgi/servers.cgi

You could also add official servers operated by the time labs of other
countries.   A list of many of them are at the end of the pdf at
https://webtai.bipm.org/ftp/pub/tai/annual-reports/bipm-annual-report/TIMESERVICES/timeservices.pdf
 .


On Wed, Aug 9, 2023, 10:30 AM Seth Mattinen via NANOG 
wrote:

> On 8/9/23 2:39 AM, Forrest Christian (List Account) wrote:
> > When GPS is working, time transmission with accuracies of under 1
> > microsecond is common.   This is especially true if the GPS integrates
> > some sort of disciplined oscillator.  Note that this is in excess of
> > what NTPd running on a typical OS can reliably retransmit.
> >
> > BUT..  if I was to choose only one protocol, it would be NTP, not GPS,
> > because of all of the reasons you mention.
> >
> > I find it distressing that sites are relying on GPS only.  I suspect
> > that this a failure to assign proper risk to using GPS.  It's
> > particularly odd when one considers that adding NTP time sources are
> > essentially free and improve robustness and reliability greatly.
> >
>
>
> I liked having a WWVB receiver in my mix, but all the hardware
> appliances (at least those offering OCXO or Rubidium oscillator options)
> seem to have rejected it in favor of GPS only. I can only conclude that
> either vendors think options like WWVB are a dead end or there's no
> demand for GPS alternatives.
>
> Products like the BlueSky GNSS Firewall exist, but not something I've
> thought was as necessary expenditure for my needs (yet). Mouser lists it
> at just under $10k.
>
> Personally I'm just not that comfortable using random unknown platform
> and unknown installation conditions time server pools over the big-I
> internet. I would possibly consider NTP servers operated by entities I
> have peering with.
>
> ~Seth
>

Re: Changes to ARIN Online - Routing Security Dashboard - RPKI & IRR integration (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[John Curran]

NANOGers -

As alluded to by Mark Kosters in his message below, we are placing on hold the 
functionality for the automatic
creation of corresponding new route objects for RPKI validated ROAs that lack 
such.  This is being done out of
an abundance of caution in order to allow us to conduct a community 
consultation in the near future to confirm
with the operational community the desired functionality in this area.

Thanks!
/John

John Curran
President and CEO
American Registry for Internet Numbers


On Aug 9, 2023, at 2:42 PM, Mark Kosters  wrote:

Responses inline starting with "MK:"

On 8/9/23, 10:21 AM, "NANOG on behalf of Jay Borkenhagen" 
mailto:arin@nanog.org> on behalf 
of j...@braeburn.org > wrote:

I agree that in ARIN's RPKI-->IRR synthesis, the set of route[6]
objects created should not depend on the maxlength used. In Mark's
phrasing, "just have least-specific."

MK: I think it was Job that brought it up as a recommendation and glad to hear 
you agree with him.

Ideally, discussion about details such as this should have occurred on
some ARIN technical mailing list in the months leading up to ARIN
deploying this change to production.

MK: I agree that we have some improvements to make here on issues that directly 
impact the operations community. In fact, we will be placing the automatic 
creation of new route objects on hold. We are in the process of issuing a 
community consultation that is being formulated to solicit feedback on several 
technical aspects of this change. We hope to hear from you and others when this 
consultation is announced (it will conducted on the arin-consult mailing list - 
https://lists.arin.net/mailman/listinfo/arin-consult).

Thanks,
Mark

Re: Changes to ARIN Online - Routing Security Dashboard - RPKI & IRR integration (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[Mark Kosters]

Responses inline starting with "MK:"

On 8/9/23, 10:21 AM, "NANOG on behalf of Jay Borkenhagen" 
mailto:arin@nanog.org> on behalf 
of j...@braeburn.org > wrote:

I agree that in ARIN's RPKI-->IRR synthesis, the set of route[6]
objects created should not depend on the maxlength used. In Mark's
phrasing, "just have least-specific."

MK: I think it was Job that brought it up as a recommendation and glad to hear 
you agree with him.

Ideally, discussion about details such as this should have occurred on
some ARIN technical mailing list in the months leading up to ARIN
deploying this change to production.

MK: I agree that we have some improvements to make here on issues that directly 
impact the operations community. In fact, we will be placing the automatic 
creation of new route objects on hold. We are in the process of issuing a 
community consultation that is being formulated to solicit feedback on several 
technical aspects of this change. We hope to hear from you and others when this 
consultation is announced (it will conducted on the arin-consult mailing list - 
https://lists.arin.net/mailman/listinfo/arin-consult).

Thanks,
Mark

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

While GPS spoofing is technically possible, all the extant spoofing only 
tampers with the ephemeris (satellite position) data, not the timing stream. 
That's because hackers have been aiming at navigation, and may not have 
expressed interest in GPS tampering when NTP tampering is so easy 🙂

To spoof GPS clocks, a hacker has to know where the antennas are, and get above 
them in order to inject a signal with the right directionality. Commercial GPS 
clock vendors have implemented various anti-spoofing measures that, for 
example, only accept signals from a certain cone of visibility, which faces up. 
They have other measures too, some of which exploit geographic diversity, so if 
 you can have two or more GPS clocks in different hub sites, the clocks will 
reject spoofing signals.

This seems like a much easier defense than deploying secure NTP (NTS), which 
adds a huge amount of complexity. At least one researcher has shown that 
poluting the existing public NTP pool with enough bogus servers to seriously 
impact network time is trivial (I cited their paper in an earlier post on this 
thread).  A well funded state actor could be laying the framework for such an 
attack as we speak, lying in wait until an opportunity to disrupt Internet NTP 
globally.

   -mel

From: NANOG  on behalf of Jay Hennigan 

Sent: Wednesday, August 9, 2023 10:58 AM
To: nanog@nanog.org 
Subject: Re: NTP Sync Issue Across Tata (Europe)

On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

> I liked having a WWVB receiver in my mix, but all the hardware
> appliances (at least those offering OCXO or Rubidium oscillator options)
> seem to have rejected it in favor of GPS only. I can only conclude that
> either vendors think options like WWVB are a dead end or there's no
> demand for GPS alternatives.

Both GPS and WWVB are over-the-air. There has been concern expressed of
a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or
spoofing WWVB is a trivial joke.

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: NTP Sync Issue Across Tata (Europe)

[Chris Adams]

Once upon a time, Jay Hennigan  said:
> Both GPS and WWVB are over-the-air. There has been concern expressed
> of a bad actor spoofing or jamming GPS. Comparatively speaking,
> jamming or spoofing WWVB is a trivial joke.

WWVB is not generally useful for precision timing applications, due to
the distance and wave reflections.  Also, from a security point of view,
I have read that it is legal to have your own low-power transmitter on
the WWVB frequency, and there are instructions for doing it with a Pi,
so it would be very cheap and easy to mess with somebody's WWVB signal.
-- 
Chris Adams 

Re: NTP Sync Issue Across Tata (Europe)

[Jay Hennigan]

On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

I liked having a WWVB receiver in my mix, but all the hardware 
appliances (at least those offering OCXO or Rubidium oscillator options) 
seem to have rejected it in favor of GPS only. I can only conclude that 
either vendors think options like WWVB are a dead end or there's no 
demand for GPS alternatives.


Both GPS and WWVB are over-the-air. There has been concern expressed of 
a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or 
spoofing WWVB is a trivial joke.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: Last Mile ISP Quality Measurements

[Mike Hammett]

Would a QoE product be able to show me that my connectivity to Slack sucks 
right now?

I've followed precinct for a long time, with Libre qos a bit less than that. I 
took it as ISP subscriber focused, not greater internet focused.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

- Original Message -
From: Dave Taht 
To: Steve Pointer 
Cc: Aaron C. de Bruyn via NANOG 
Sent: Wed, 09 Aug 2023 07:36:19 -0500 (CDT)
Subject: Re: Last Mile ISP Quality Measurements

I am reluctant to respond because it might end up sounding like an ad
for libreqos.io.

Leaving aside the tcp rtt tracking, the cake shaping, the mark and
drop statistics in that product, the (mostly wireless) ISPs we work
with typically have a dashboard of long term SNMP statistics of key
parameters like signal strength (RSSI), a heatmap of rtts to those
routers, and the upstreams, (smokeping cannot handle this kind of
density), a bandwidth tracker (usually on a 5 minute interval), a few
raspberry pi or equivalents at the towers doing active measurements on
demand, and a set of actionable items derived from that that the
support techs work off of.
Then there is a per customer screen that captures as much as possible
useful about the customer and every hop along the way.

There are a lot of pics of dashboards like this on the web, see
preseem, paraqum, and of course libreqos for examples. People use a
variety of backend products for it (grafina, redis, influx are
popular).

hope this helps.

On Wed, Aug 9, 2023 at 5:23 AM Steve Pointer  wrote:
>
> Have you considered hosting a Ripe Anchor?
>
> https://atlas.ripe.net/anchors/about/
>
> Minimal cost, good of the Internet project, good insights, answers the use 
> case you describe.
>
> Steve P
>
>


-- 
Podcast: https://www.youtube.com/watch?v=bxmoBr4cBKg
Dave Täht CSO, LibreQos

Re: Last Mile ISP Quality Measurements

[Mike Hammett]

I have a probe, but not an anchor.

That would just help with simple reachability issues to probes that test 
against it, wouldn't it? It wouldn't necessarily be able to monitor popular 
Internet destinations or across different peers?



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

- Original Message -
From: Steve Pointer 
To: Aaron C. de Bruyn via NANOG 
Sent: Wed, 09 Aug 2023 04:49:02 -0500 (CDT)
Subject: Re: Last Mile ISP Quality Measurements

Have you considered hosting a Ripe Anchor?

https://atlas.ripe.net/anchors/about/

Minimal cost, good of the Internet project, good insights, answers the use case 
you describe.

Steve P

VIDEO - NANOG Talks to Seattle Community Network + Guest Column: Geoff Huston

[Nanog News]

*VIDEO - NANOG Talks to Seattle Community Network*
*NANOG Talks to SCN About Connecting the Underserved Across Puget Sound*

*The Seattle Community Network (SCN) is an Internet built by the community
for the community.*

NANOG Executive Director Edward McNair sat down with a volunteer of SCN +
Ph.D. candidate in the Department of Computer Science at the University of
Washington, Esther Jang, at our most recent meeting, NANOG 88 (Seattle, Wa,
12 - 14 Jun) to discuss the mission of SCN and how they are connecting +
empowering underserved regions locally.

*WATCH NOW *


*Guest Column: Geoff Huston "The Source of Time, According to a
Technologist"*
*Exploring the Definition of Time from Early Astronomical Observations to
its use in the Internet Today*

An underappreciated aspect of our digital infrastructure is how dependent
we are on time.

Disrupting time leads to communication disruption and can result in various
forms of service compromise. A synchronized view of the time is essential
to the Internet's service platform.

*READ MORE *


*Upcoming Fall Online Course*
*Fundamentals of Designing and Deploying Computer Networks*

*Date: *11 September –  20 October 2023
*Title:* Fundamentals of Designing and Deploying Computer Networks

*Description:* This course will discuss the fundamentals of networking,
Ethernet, and WIFI technologies. It will additionally teach the planning,
design, and deployment of simple LANs and cover how to connect a LAN to the
Internet. The course will also present the most common ways to connect a
LAN to the Internet (Mobile Internet, ADSL, Fiber) and how to set up the
connections.

*MORE INFO * 

*Video of the Week*
*Deploying a Backbone in APAC with Pierre-Yves Maunier*

Maunier discusses what his team learned and changed during their multi-year
journey expanding their network footprint in the Asia Pacific region.

He will speak about the challenges they faced, their mistakes, and how they
worked around them with various iterations of their deployment.

*WATCH NOW * 

Re: NTP Sync Issue Across Tata (Europe)

[Seth Mattinen via NANOG]

On 8/9/23 2:39 AM, Forrest Christian (List Account) wrote:
When GPS is working, time transmission with accuracies of under 1 
microsecond is common.   This is especially true if the GPS integrates 
some sort of disciplined oscillator.  Note that this is in excess of 
what NTPd running on a typical OS can reliably retransmit.


BUT..  if I was to choose only one protocol, it would be NTP, not GPS, 
because of all of the reasons you mention.


I find it distressing that sites are relying on GPS only.  I suspect 
that this a failure to assign proper risk to using GPS.  It's 
particularly odd when one considers that adding NTP time sources are 
essentially free and improve robustness and reliability greatly.





I liked having a WWVB receiver in my mix, but all the hardware 
appliances (at least those offering OCXO or Rubidium oscillator options) 
seem to have rejected it in favor of GPS only. I can only conclude that 
either vendors think options like WWVB are a dead end or there's no 
demand for GPS alternatives.


Products like the BlueSky GNSS Firewall exist, but not something I've 
thought was as necessary expenditure for my needs (yet). Mouser lists it 
at just under $10k.


Personally I'm just not that comfortable using random unknown platform 
and unknown installation conditions time server pools over the big-I 
internet. I would possibly consider NTP servers operated by entities I 
have peering with.


~Seth

Re: Changes to ARIN Online - Routing Security Dashboard - RPKI & IRR integration (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[Jay Borkenhagen]

Job Snijders via NANOG writes:
 > >
 > > > Would it not be advantageous to create at a minimum the 256 of the
 > > > 'least-specific' objects?
 > > 
 > > MK: That may be a reasonable approach. Do you see any adverse effects
 > > in simplifying the IRR Route creation logic to just have
 > > least-specific?
 > 
 > I don't think I see a downside of mapping a single VRP to a single IRR
 > route/route6 object.
 > 

I agree that in ARIN's RPKI-->IRR synthesis, the set of route[6]
objects created should not depend on the maxlength used.  In Mark's
phrasing, "just have least-specific."

Ideally, discussion about details such as this should have occurred on
some ARIN technical mailing list in the months leading up to ARIN
deploying this change to production.

Thanks.

Jay B.

Re: Last Mile ISP Quality Measurements

[Tom Beecher]

>
> measure the quality of their connections


Really depends on what you are trying to measure. Some metrics are going to
be great at telling you the quality and performance of the network at L3,
but thanks to the Stupid Content Provider Tricks that we use, won't tell
you anything about the L4/L7 experience that your customers may have.

Tons of different things you could measure, it's just putting them together
in such a way that all parties in the conversation have the same context
and understanding that can be exceptionally tricky.



On Tue, Aug 8, 2023 at 5:09 PM Mike Hammett  wrote:

> What are other last mile ISPs doing to measure the quality of their
> connections? We all know pinging various destinations. We also all know
> that pinging a destination doesn't necessarily tell you the whole quality
> story.
>
> I currently have Smokeping pulling the HTTPS for about 20 - 25 of the
> "top" websites, per the old Alexa rankings. I feel as though I could be
> doing more. I am more closely wanting to emulate the end-user experience in
> a repeatable, quantifiable fashion. I'd like to do A/B comparisons as well.
> When I make X change, how does it change?
>
>
> If I'm already doing the low-hanging fruit path, then so be it.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
>

Re: Last Mile ISP Quality Measurements

[Dave Taht]

I am reluctant to respond because it might end up sounding like an ad
for libreqos.io.

Leaving aside the tcp rtt tracking, the cake shaping, the mark and
drop statistics in that product, the (mostly wireless) ISPs we work
with typically have a dashboard of long term SNMP statistics of key
parameters like signal strength (RSSI), a heatmap of rtts to those
routers, and the upstreams, (smokeping cannot handle this kind of
density), a bandwidth tracker (usually on a 5 minute interval), a few
raspberry pi or equivalents at the towers doing active measurements on
demand, and a set of actionable items derived from that that the
support techs work off of.
Then there is a per customer screen that captures as much as possible
useful about the customer and every hop along the way.

There are a lot of pics of dashboards like this on the web, see
preseem, paraqum, and of course libreqos for examples. People use a
variety of backend products for it (grafina, redis, influx are
popular).

hope this helps.

On Wed, Aug 9, 2023 at 5:23 AM Steve Pointer  wrote:
>
> Have you considered hosting a Ripe Anchor?
>
> https://atlas.ripe.net/anchors/about/
>
> Minimal cost, good of the Internet project, good insights, answers the use 
> case you describe.
>
> Steve P
>
>


-- 
Podcast: https://www.youtube.com/watch?v=bxmoBr4cBKg
Dave Täht CSO, LibreQos

Re: Last Mile ISP Quality Measurements

[Steve Pointer]

Have you considered hosting a Ripe Anchor?

https://atlas.ripe.net/anchors/about/

Minimal cost, good of the Internet project, good insights, answers the use case 
you describe.

Steve P

Re: NTP Sync Issue Across Tata (Europe)

[Masataka Ohta]

John Gilmore wrote:


 I was also speaking specifically about installing GPS antennas in
 viable places, not using a facility-provided GPS or NTP service.


Am I confused?  Getting the time over a multi-gigabit Internet from a
national time standard agency such as NIST (or your local country's
equivalent) should produce far better accuracy and stability than
relying on locally received GPS signals.


When the (wrong) question is "how to build a stratum 1 server?",
that can not be an answer.


GPS uses very weak radio
signals which are regularly spoofed by all sorts of bad actors:


The question, seemingly, is not "how to build a secure stratum
1 server?".

BTW, the proper question should be "how to obtain secure time?".

Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

[Forrest Christian (List Account)]

When GPS is working, time transmission with accuracies of under 1
microsecond is common.   This is especially true if the GPS integrates some
sort of disciplined oscillator.  Note that this is in excess of what NTPd
running on a typical OS can reliably retransmit.

BUT..  if I was to choose only one protocol, it would be NTP, not GPS,
because of all of the reasons you mention.

I find it distressing that sites are relying on GPS only.  I suspect that
this a failure to assign proper risk to using GPS.  It's particularly odd
when one considers that adding NTP time sources are essentially free and
improve robustness and reliability greatly.

NTP is not without it's risks but the most common server implementation is
specifically designed to be able to discard time sources which are not
telling the truth, provided the server is given enough valid time sources.
Even if a spoofed or misconfigured server is giving the wrong time,  NTPd
will be able to ignore those errant time sources.

 When configured with numerous network time sources and a GPS source,  NTPd
will determine what the correct time should be, and then will use the
higher accuracy GPS source to improve the overall accuracy.  This is more
or less automatic since the latency to the GPS time source will be
essentially zero when compared to a typical network source.

However,  if the GPS source starts lying about the time,  NTPd will start
ignoring it as a potential time source even with the lower latency.
Without having non-GPS sources in your configuration, this essentially free
protection against GPS spoofing is no longer available since it has nothing
to compare it to.

If your network is large enough that you could install multiple GPS
receivers in diverse locations,  then I'd configure all of the NTPd servers
to pull from all of the GPS receivers.  That way you gain additional
redundancy.  I'd still not drop the public trusted NTP servers though.




On Tue, Aug 8, 2023, 2:58 PM John Gilmore  wrote:

> > I was also speaking specifically about installing GPS antennas in
> > viable places, not using a facility-provided GPS or NTP service.
>
> Am I confused?  Getting the time over a multi-gigabit Internet from a
> national time standard agency such as NIST (or your local country's
> equivalent) should produce far better accuracy and stability than
> relying on locally received GPS signals.  GPS uses very weak radio
> signals which are regularly spoofed by all sorts of bad actors:
>
>   https://www.gps.gov/spectrum/jamming/
>
> for all sorts of reasons (like misleading drone navigation):
>
>   https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident
>
> Depending on satnav systems creates a large single point of failure for
> worldwide civilian infrastructure.
>
> Jamming GPS with subtly fake time data near big data centers seems like
> an easy move that would cause all sorts of distributed algorithms to
> start failing in unusual ways.  And in a more serious wartime attack,
> many or most GPS satellites themselves would be destroyed or disabled.
> Yet digital radio modulations like FT8 or DMR rely on tight time
> synchronization among different transmitters.  So do many modern
> cellphone modulations -- not to mention distributed database sync
> algorithms.  Depending on any of these for emergency communications when
> their time comes from GPS, is a recipe for having no communications
> during wars or cyber-wars in which GPS satellites are attacked or
> jammed.  See a longer explanation here:
>
>   https://www.ardc.net/apply/grants/2020-grants/grant-ntpsec/
>
> I suspect that even today, if you rely on civilian GPS time near the US
> White House, Pentagon, or other military targets like bases, you will
> discover "anomalies" in the local radio GPS data, compared to what you
> get from an authenticated time standard over NTP.  How reliable is
> civilian GPS time in Ukraine these days?
>
> John
>
>

Re: Last Mile ISP Quality Measurements

[touseef.rehman1--- via NANOG]

 I would personally have end IT friendly mimick and test their existing 
systems on the nee ISP. Especially cloud tech having read a book on 
cloud by some PHD administrators they redefined  cloud as being cloudy 
in a sense to do with not knowing what route your clients packets will 
take to reach the server.






Sent via BT Email App



From: Tim Burke 

Sent: 8 August 2023 22:47:06 BST

To: Mike Hammett , NANOG 

Subject: Re: Last Mile ISP Quality Measurements






   We are doing something similar with netpath in Solarwinds, but 
mainly using the stream URLs of some popular streaming services that we 
see commonly used (FuboTV, etc). Came in handy recently in tracking down 
customer complaints that ended up being a peering capacity issue further 
upstream.













   Tim















  From: NANOG  on behalf of Mike 
Hammett 

 Sent: Tuesday, August 8, 2023 4:08 PM
 To: NANOG 
 Subject: Last Mile ISP Quality Measurements


    








What are other last mile ISPs doing to measure the quality of their 
connections? We all know pinging various destinations. We also all know 
that pinging a destination doesn't necessarily tell you the whole 
quality story.









I currently have Smokeping pulling the HTTPS for about 20 - 25 of 
the "top" websites, per the old Alexa rankings. I feel as though I could 
be doing more. I am more closely wanting to emulate the end-user 
experience in a repeatable, quantifiable fashion. I'd like to do A/B 
comparisons as well. When I make X change, how does it change?

















If I'm already doing the low-hanging fruit path, then so be it.










 -

 Mike Hammett

 Intelligent Computing Solutions 

   
 
 



 Midwest Internet Exchange 

   
 



 The Brothers WISP 

   

Looking for a Telus cellular last mile facilities operations contact

[Eric Kuhnke]

I have observed a Telus cellular site shelter that's making a terrible, not
normal ventilation noise.

It's a 12+ foot length prefab assemble on site shelter located in the
basement parking garage of a 23 floor tower in downtown Vancouver. I know
what this POP's normal ventilation sounds like, having seen.and heard it in
operation for several years now, and this isn't healthy.

Based on the amount of sectors on the roof and ptp microwave links from
this site I would guess this is a more important than ordinary site, so if
Telus doesn't want to wait for it to overheat and die, contact me off list
for the street address of the building.

I have already tried spending about 45 minutes on the phone with Telus
customer service (I am not a customer) and this has been making the "I am a
dying ventilation blower" noise for two weeks now.