NTP Performance

2019-11-20 Thread Richard Laager via devel
I now have a second stratum 1 server, with an independent setup. This allows me to compare the two. Why does ntp1 have this very specific repeating pattern of local clock offset? It's roughly +7 us, -5 us, +2 us, -4 us and then repeats again, over and over. We can also see that in the histogram, wh

Re: NTP Performance

2019-11-20 Thread Paul Theodoropoulos via devel
Just a shot in the dark, but it looks to me like an errant overly-greedy cron job or systemd timer - perhaps something else is polling the serial port on a schedule. If you set up hourly graphs you might be able to correlate it with something else firing on the server at those intervals. Unfor

Re: NTP Performance

2019-11-20 Thread Gary E. Miller via devel
Yo Richard! On Wed, 20 Nov 2019 16:02:32 -0600 Richard Laager via devel wrote: > I now have a second stratum 1 server, with an independent setup. This > allows me to compare the two. Cool. > Why does ntp1 have this very specific > repeating pattern of local clock offset? It's roughly +7 us, -5

Re: NTP Performance

2019-11-20 Thread Richard Laager via devel
On 11/20/19 4:33 PM, Paul Theodoropoulos via devel wrote: > Just a shot in the dark, but it looks to me like an errant > overly-greedy cron job or systemd timer - perhaps something else is > polling the serial port on a schedule. If you set up hourly graphs you > might be able to correlate it with

Re: NTP Performance

2019-11-20 Thread Hal Murray via devel
>> Also check the ntpd is locked to your PPS during the events. > Is there a way to check that retroactively? Turn on protostats and/or the flags that lots of stuff into ntp.log logconfig =syncall +clockall +peerall +sysall 18 Nov 12:32:20 ntpd[140066]: PROTO: 192.168.1.32 f4da 8a sys_peer 1

Re: NTP Performance

2019-11-20 Thread ASSI via devel
Richard Laager via devel writes: > I now have a second stratum 1 server, with an independent setup. This > allows me to compare the two. Why does ntp1 have this very specific > repeating pattern of local clock offset? It's roughly +7 us, -5 us, +2 > us, -4 us and then repeats again, over and over.

Re: NTP Performance

2019-11-20 Thread Hal Murray via devel
> Why does ntp1 have this very specific repeating pattern of local clock > offset? It's roughly +7 us, -5 us, +2 us, -4 us and then repeats again, over > and over. That's pretty weird. > ntp1 uses the PPS refclock with a polling interval of 3 from a telecom GPSDO > (OCXO): What brand/model?

Re: NTP Performance

2019-11-22 Thread Richard Laager via devel
On 11/21/19 12:55 AM, ASSI via devel wrote: >> I could try fiddling around with the polling interval. Next steps might >> be to try raising the polling the interval to 4 and/or lowering it to 1. > It is generally inadvisable to use too short polling intervals unless > rthe clock you are disciplini

Re: NTP Performance

2019-11-22 Thread Richard Laager via devel
On 11/21/19 1:15 AM, Hal Murray wrote: >> ntp1 uses the PPS refclock with a polling interval of 3 from a telecom GPSDO >> (OCXO): > What brand/model? Symmetricom TimeSource 3000. It's an older, end-of-life model that we've had in place for a while. We have four of them in different sites, but thi

Re: NTP Performance

2019-11-22 Thread ASSI via devel
Richard Laager via devel writes: > On 11/21/19 12:55 AM, ASSI via devel wrote: >>> I could try fiddling around with the polling interval. Next steps might >>> be to try raising the polling the interval to 4 and/or lowering it to 1. > >> It is generally inadvisable to use too short polling intervals

Re: NTP Performance

2019-11-22 Thread ASSI via devel
Richard Laager via devel writes: > On 11/21/19 1:15 AM, Hal Murray wrote: >>> ntp1 uses the PPS refclock with a polling interval of 3 from a telecom GPSDO >>> (OCXO): >> What brand/model? > > Symmetricom TimeSource 3000. It's an older, end-of-life model that we've > had in place for a while. We ha

Re: NTP Performance

2019-11-22 Thread Richard Laager via devel
On 11/23/19 1:28 AM, ASSI via devel wrote: > Richard Laager via devel writes: >> On 11/21/19 1:15 AM, Hal Murray wrote: ntp1 uses the PPS refclock with a polling interval of 3 from a telecom GPSDO (OCXO): >>> What brand/model? >> >> Symmetricom TimeSource 3000. It's an older, end-o

Re: NTP Performance

2019-11-22 Thread Richard Laager via devel
On 11/23/19 12:57 AM, ASSI via devel wrote: > Richard Laager via devel writes: >> On 11/21/19 12:55 AM, ASSI via devel wrote: I could try fiddling around with the polling interval. Next steps might be to try raising the polling the interval to 4 and/or lowering it to 1. >> >>> It is gener

Re: NTP Performance

2019-11-23 Thread Richard Laager via devel
Also, what does this mean and is it a problem (it's an ERR level)? I'm seeing it on both servers. 2019-11-23T01:49:33.497786-06:00 ntp1 ntpd[28568]: CLOCK: ts_prev 1574495373 s + 497394102 ns, ts_min 1574495373 s + 497388500 ns 2019-11-23T01:49:33.497936-06:00 ntp1 ntpd[28568]: CLOCK: ts 15744953

Re: NTP Performance

2019-11-23 Thread Hal Murray via devel
Also, what does this mean and is it a problem (it's an ERR level)? I'm seeing it on both servers. 2019-11-23T01:49:33.497786-06:00 ntp1 ntpd[28568]: CLOCK: ts_prev 1574495373 s + 497394102 ns, ts_min 1574495373 s + 497388500 ns 2019-11-23T01:49:33.497936-06:00 ntp1 ntpd[28568]: CLOCK: ts 15744953

Re: NTP Performance

2019-11-23 Thread Achim Gratz via devel
[Hal, I've asked before, but again: Can you please configure whatever you use as your mailer to not always break threading when you reply?] Hal Murray via devel writes: > The code that reads the clocks works hard to make sure that fuzzing the > bottom > bits doesn't make time go backwards. T

Re: NTP Performance

2019-11-23 Thread Gary E. Miller via devel
Yo Richard! On Sat, 23 Nov 2019 01:53:33 -0600 Richard Laager via devel wrote: > That trade-off makes sense to me. But, of the two, it seems like > following the PPS closely is what I'd want. The PPS is supposed to be > more accurate than my computer. There is a gpsd program in the contrib/ dir

Re: NTP Performance

2019-11-23 Thread Richard Laager via devel
On 11/23/19 8:19 PM, Gary E. Miller via devel wrote: > There is a gpsd program in the contrib/ directory. It tests your > CPU granularity. On a Raspberry Pi that is about 52 ns. Worse > on an Intel chip. It looks like you're talking about clock_test.c. I grabbed the version from gpsd git: https

Re: NTP Performance

2019-11-23 Thread Richard Laager via devel
On 11/23/19 3:02 AM, Hal Murray wrote: > The code that reads the clocks works hard to make sure that fuzzing the > bottom > bits doesn't make time go backwards. That logging is what happens when > things > go wrong. > > What sort of hardware/OS are you running on? 2x Intel Xeon CPU X5460 @ 3

Re: NTP Performance

2019-11-23 Thread ASSI via devel
Gary E. Miller via devel writes: > There is a gpsd program in the contrib/ directory. It tests your > CPU granularity. On a Raspberry Pi that is about 52 ns. Worse > on an Intel chip. The actual granularity on RasPi can't be better than 52ns (the clock it's based on is 19.2MHz) and you can dete

Re: NTP Performance

2019-11-24 Thread Richard Laager via devel
On 11/23/19 1:53 AM, Richard Laager via devel wrote: > > On ntp2 (again, this is the u-blox 6 eval kit), I'm getting duplicated > sequence numbers, which doesn't seem quite right: > > rlaager@ntp2:~$ sudo ppstest /dev/pps0 > trying PPS source "/dev/pps0" > found PPS source "/dev/pps0" > ok, found

Re: NTP Performance

2019-11-24 Thread Achim Gratz via devel
Richard Laager via devel writes: > Okay, so upon further reflection, the timestamps seem to indicate that I > need to use the clear/falling edge. This also explains the duplication, > I think. This is despite the GPS being configured for rising edge. So as > Hal (I think) said, something is inverti

Re: NTP Performance

2019-11-24 Thread Gary E. Miller via devel
Yo ASSI! On Sun, 24 Nov 2019 07:37:13 +0100 ASSI via devel wrote: > Gary E. Miller via devel writes: > > There is a gpsd program in the contrib/ directory. It tests your > > CPU granularity. On a Raspberry Pi that is about 52 ns. Worse > > on an Intel chip. > > The actual granularity on Ra

Re: NTP Performance

2019-11-24 Thread Gary E. Miller via devel
Yo Richard! On Sat, 23 Nov 2019 23:08:06 -0600 Richard Laager via devel wrote: > It looks like you're talking about clock_test.c. > Below are ten runs from ntp1. > min 39 ns, max 120 ns, mean 89 ns, median 96 ns, StdDev 18 ns A min/max spread of almost 100 ns. Now you see why I say the NEO-M

Re: NTP Performance

2019-11-25 Thread Achim Gratz via devel
Richard Laager via devel writes: > These both have the following CPU (which is older): > Intel(R) Xeon(R) CPU X5460 @ 3.16GHz These may not yet have consistent TSC between cores/sockets (or require BIOS tweaks for that). The other result where you show that MONOTONIC_RAW reads much slo

Re: NTP Performance

2019-11-25 Thread Richard Laager via devel
I have logs going back to 2019-10-26. These clock fuzz errors started on ntp1 on 2019-11-02 and ntp2 on 2019-11-21. On 2019-11-02, I upgraded to NTPsec 1.1.7 (from 1.1.3) and enabled NTS (as both a client and server). On 2019-11-08, I added the GPS to ntp2. Based on the dates, that seems unrelate

Re: NTP Performance

2019-11-26 Thread Achim Gratz via devel
Richard Laager via devel writes: >> These may not yet have consistent TSC between cores/sockets (or require >> BIOS tweaks for that). > /proc/cpu says constant_tsc, but that's it (besides "tsc", of course). > That is, I do _not_ have nonstop_tsc, so therefore I presume I do not > have the "invarian

Re: NTP Performance

2019-11-27 Thread Richard Laager via devel
On 11/26/19 12:19 PM, Achim Gratz via devel wrote: > Richard Laager via devel writes: >> I only have the clock fuzzing errors on my NTP servers. I don't have an >> exact matching configuration that's not an NTP server, but: Similar >> hardware running Debian 10 and ntpsec 1.1.3 does not. Two eras o

Fuzzing Messages (was: NTP Performance)

2019-11-30 Thread ASSI via devel
Richard Laager via devel writes: > Also, what does this mean and is it a problem (it's an ERR level)? I'm > seeing it on both servers. > > 2019-11-23T01:49:33.497786-06:00 ntp1 ntpd[28568]: CLOCK: ts_prev 1574495373 > s + 497394102 ns, ts_min 1574495373 s + 497388500 ns > 2019-11-23T01:49:33.49793