[time-nuts] Re: NIST NTP servers way off for anyone else?

[Trent Piepho]

On Tue, Dec 14, 2021 at 2:21 PM Hal Murray  wrote:
>
> > I've seen cards (ethtool) that support several time options - what are  they
> > and how do I use them?
>
> I'm not sure which options you are referring to.

Probably the flags from ethtool -T output:

Time stamping parameters for eth0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
hardware-receive  (SOF_TIMESTAMPING_RX_HARDWARE)
software-receive  (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
hardware-raw-clock(SOF_TIMESTAMPING_RAW_HARDWARE)

The software ones are normally alway present and are what you describe
with the kernel's timestamping.  The hardware are done by the network
PHY and need 1588 support in the driver, which is not common.  I don't
think any of the RPis support it.

>
> You can't have boxes on the internet update packets if you are interested in
> security.

That seems too restrictive.  Consider that TLS doesn't include the
TCP/IP header, which can be modified by IP fragmentation, and that is
still considered secure.

I think one could design a protocol such that each appended timestamp
is signed, and included a digest of all timestamps before it, so that
while one does not trust every timestamp in the chain, it can be
trusted that each timestamp was generated by entity that said it
generated it and that any timestamps generated by a trusted entity
were not later modified by a untrusted one.
___
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an 
email to time-nuts-le...@lists.febo.com
To unsubscribe, go to and follow the instructions there.

[time-nuts] Re: PPS latency? User vs kernel mode

[Trent Piepho]

On Mon, Dec 13, 2021 at 8:03 AM David Taylor via time-nuts
 wrote:
>
> On 13/12/2021 04:17, Adam Space wrote:
> >> What do you mean by "kernel mode"?
> > I am referring to this guide
> >   that another user here
> > recommended to me a bit ago. I am not too sure myself since I am relatively
> > new to this.
>
> That document is now historic background.
>
> It refers to the early "kernel mode" when (AIUI) part of the OS rather than 
> NTP
> did the fine clock adjustment.  I recall that early Raspberry PI OS kernels 
> did
> not include all the functions required for the most accurate timekeeping, and
> that a kernel recompile was required.

For kernel mode, there are really two things I can think of that might
be referred to.  This is stated well in the ntp.org FAQ:

5.2.4.2. [...] two mechanisms: Capturing an external event with high
accuracy, and applying that event to the clock model.

In Linux, we have pps-gpio for the first of those.  There are also
other pps clients, e.g. serial port CD line, but the gpio is the
common one.  This timestamps the PPS pulse in the kernel, which avoids
a number of significant sources of error.

Linux gained a RFC2783 kernel PPS system in 2009 and the pps-gpio
client was added in 2011.  Prior to this, kernel pps timestamping
required patching the kernel to add separately maintained
implementations.

But even with pps-gpio in the kernel in 2011, configuring it was
difficult.  The dynamic device tree overlay support used on modern RPi
distros didn't exist.  pps-gpio didn't even support device tree based
configuration.  Configuration was done by writing C code in the kernel
to create platform_data that configured various things, pps driver,
pinmux, gpio controller, etc.  This was trivial for embedded software
developers and kernel hackers, but put it completely out of the reach
of many.

Prior to kernel pps support, the common way to get a PPS was done in
userspace.  The PPS was connected to the serial port CD line and
userspace code would wait for a DCD interrupt via the TIOCMIWAIT tty
ioctl.  This has significantly more error than the above kernel
timestamping method.  But one did not need any sort of kernel
configuration to use it and Linux has supported UART DCD interrupts
and TIOCMIWAIT for a very long time (older than the history in git,
likely in the early 90s.)

Now we can consider the 2nd part, applying that event to the clock model.

In the beginning, we had the adjtime() call.  This adds an offset to
the clock, which the kernel applies gradually over a period of time.
This offset must be repeatedly calculated as the clock's phase error
drifts from zero and then reprogrammed via control loop.  I.e., the
PLL or FLL is implemented in userspace (ntpd).

Then we gained adjtimex() (also ntp_adjtime() and clock_adjtime())
from David Mills starting in 1993 and leading up to RFC 5905 in 2010.
This allows the PLL/FLL to be implemented in the kernel.  But it's
still controlled from ntpd.  ntpd does the actual network protocol
and/or gets PPS timestamps and steers the PLL via adjtimex calls.
Here we have ntpd opens /dev/pps0, receives timestamps, and calculates
values to steer a kernel PLL via adjtimex calls.

And finally we can have the kernel act on the PPS timestamps itself.
But NTP network traffic might be a bridge too far.  Mills describes
this as a "hardpps()".  This was added to Linux in 2011 by Alexander
Gordeev, based on Mills' work, but with a different implementation.
This is a "kernel consumer" of PPS timestamps, which can be used to
steer the PLL entirely inside kernel code.  It's enabled with the
kernel config option NTP_PPS.  One can easily see if this is present
in the kernel via the usual methods.  It is often NOT enabled because
it doesn't work with tickless kernels.  I do now know if current ntpd
supports this or what it takes to use it with ntpd.  I have used it,
but I wrote my own code from scratch to integrate it with a GPS module
to perform time sync without installing either ntpd nor chrony.
___
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an 
email to time-nuts-le...@lists.febo.com
To unsubscribe, go to and follow the instructions there.

[time-nuts] Re: PPS latency? User vs kernel mode

[Trent Piepho]

On Mon, Dec 13, 2021 at 6:56 AM Poul-Henning Kamp  wrote:
>
> In FreeBSD you get all that for free:
>
> https://papers.freebsd.org/2002/phk-timecounters.files/timecounter.pdf
>

"It is painfully obvious that the interrupt latency is the dominant
noise factor in PPS timestamping in the second case."

This is exactly what I found when I did this 15 years after your
paper.  Though what I think is more interesting, is that despite using
15 years newer hardware, the resulting PPS precision ends up being
nearly the same for both timestamping methods.  Your final graphs are
very similar the one I have tried (succeeded?) at posting, other than
you using a time series I used a histogram.
___
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an 
email to time-nuts-le...@lists.febo.com
To unsubscribe, go to and follow the instructions there.

[time-nuts] Re: PPS latency? User vs kernel mode

[Trent Piepho]

I found the plot I made earlier (Aug 2017), which I've attached.  The
link I sent in 2017 and in the list archive still works, but I'm not
allowed to post links to google photos anymore.

The uncorrected value would be comparable to pps-gpio, without a
hardware timer.  Worst case is 100 µs.  This was an Altera Cyclone V,
dual core Cortex-A9 at about 700 MHz.  Current RPi generation is
considerably faster, but CPU speed is not really the issue here.

IIRC, pps-gpio creates the timestamp in hard irq context.  This means
it will preempt any tasks, both those running in user mode and in
kernel mode.  It should only be delayed, software wise, by another
hard irq handler or when local irqs have been disabled on that core
for a spinlock.  The code in pps-gpio should be implemented so that
there is no contention for shared software resources, such as a global
lock to read the kernel clock.

There are also hardware based sources of error, such as contention on
the AXI bus to access main memory (but not the local CPU cache).  But
these should be much smaller than 500 µs.

So your culprit is a driver that spends a long time in hard irq
context or holds a spinlock for a long time.  The worst culprits are
invariably GPU drivers.

> I recall, I tried also to limit timing relevant tasks to a separate CPU core
> (this helped a bit, but not really enough), and I am not completely sure
> whether fiddling around with the timer IRQ registers of the interrupt
> controller really disabled all timer IRQs on that core completely, I just
> remember that the documentation on how to write the correct dts-files for
> doing so was close to non-existent, so I might have done that incorrectly...

I have yet to see steering interrupt to a particular core work on ARM.
But it has been a while since I last tried.  You must ensure your IRQ
is delivered to one set of core(s), while any ill-behaving IRQ is not.
And must also ensure any calls from userspace into kernel mode (ioctl,
read, write, and so on) that will be handled by an ill-behaved driver
are also not run on those cores.  So it is not just the steering of
hardware IRQs one must do, but also CPU affinity sets for userspace
tasks and kernel threads.

I suggest looking at LTTng to trace the kernel.  You can generate a
timeline of every irq, interrupt masking, system call, and scheduler
decision.  You can then search for a trace event where the pps-gpio
phase offset was > 100 µs and see which CPU handled the hard irq and
then exactly what that CPU was doing that led up to it taking so long
to get around to making that timestamp.

> Without a hardware time stamper for inputs and a programmable hardware timer
> for outputs, I would not trust the RPi for timing purposes.

With the GPU disabled, it should be good enough for many use cases.
Certainly, if you are writing something in Python, it will not be a
limiting factor!  But of course a hardware timestamper is orders of
magnitude better.  It is a shame that hardware seems so rare in SoCs.
___
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an 
email to time-nuts-le...@lists.febo.com
To unsubscribe, go to and follow the instructions there.

[time-nuts] Re: PPS latency? User vs kernel mode

[Trent Piepho]

What I did was measure the phase offset of the PPS timestamps in Linux,
after ntp has locked the clock to the PPS.  I think would be made up of
three main sources of error:

1. The phase offset in the PPS signal itself.
2. The error in the Linux clock doing the timestamping, over a tau of 1
second.
3. The jitter in the latency of the timestamp's creation after the pulse.

Of these, I think it's safe to assume that 3 is by far the greatest.  And
at the very least we get an upper bound for that error.

I think you can find some graphs I made in the list archives.  Switching
from kernel GPIO PPS timestamping to a kernel driver for a hardware
timestamper was a vast improvement.  I didn't even bother with userspace
timestamping, it would surely be far worse than kernel mode, having all the
same sources of error kernel mode does plus several significant other
sources.

I don't think any of the RPi boards have a SoC with hardware timestamp.
They might have IEEE1588 hardware in the ethernet PHY, but it seems
Broadcom won't release docs to enable support for that.  So the best you
can do without custom hardware would be the pps-gpio module.  This looks
quite easy to setup, with dt-overlays already created.  Just adding:

dtoverlay=pps-gpio,gpiopin=4

Should create a pps device on gpio pin 4 that shows up as /dev/pps0 and can
be used directly by e.g. chrony.  There's a ntp-like algorithm in the
kernel that can be used to sync the kernel clock to a PPS without using
ntpd or chrony or anything in userspace.   It needs to be enabled the
kernel config and then turned on via adjtimex() calls.

On Sat, Dec 11, 2021 at 4:01 PM Adam Space  wrote:

> I have a PPS with Raspberry Pi setup going. Of course, in terms of
> precision, it's quite good (for me it's as good as I'd ever care for), and
> for accuracy it's quite good too. Although I guess the problem is, I don't
> really know how good the accuracy is, nor am I sure how I would go about
> finding out.
>
> The only factor here would be a delay between the PPS signal and the
> processing in the Pi. I am not using kernel mode right now, which I assume
> makes this even worse. Nonetheless, it is certainly no more than one or two
> ms. When I run ntp with several other peers, you can't tell much of a
> difference. The accuracy limits of NTP are greater than the potential delay
> introduced by the processing of the PPS signal, so even running NTP with a
> lot of peers and averaging over long periods of time doesn't give much info
> on this. (Perhaps if I ran it long enough I could get the mean offset, but
> I think the uncertainty on this value would be larger than the value
> itself).
>
> I am curious what other folks with this type of set-up have done. Just
> leave it at that and accept that there's some unknown latency? Or perhaps
> the latency is on the order of microseconds, in which case it's not really
> a big deal for me.
>
> With regards to kernel mode, I was looking to give it a try. However, the
> guides I've found on this, some posted several years ago on this list, are
> pretty outdated. If someone has a suggestion for this, I'm interested in
> hearing it. Also, my Raspberry Pi is relatively new, running Bullseye, if
> that matters.
>
> Adam
> ___
> time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send
> an email to time-nuts-le...@lists.febo.com
> To unsubscribe, go to and follow the instructions there.
>
___
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an 
email to time-nuts-le...@lists.febo.com
To unsubscribe, go to and follow the instructions there.

[time-nuts] Re: When did computer clocks get so bad?

[Trent Piepho]

On Wed, Sep 29, 2021 at 1:48 PM Poul-Henning Kamp  wrote:
> Since these embedded chips generally are incredibly robust with
> respect to timing, the xtal on the BOM is the cheapest that will
> meet spec.

Crystals?  We don't need no stinking crystals!

At least for the RTC.  Chips like, e.g. the NXP iMX6, can use an
internal ring oscillator in place of the customary 32kHz crystal to
save even more on BOM cost and board space.  Still needs a faster xtal
for full operation, but low power mode and RTC can use the ring
oscillator and keep the 24 MHz xtal powered down.
___
time-nuts mailing list -- time-nuts@lists.febo.com -- To unsubscribe send an 
email to time-nuts-le...@lists.febo.com
To unsubscribe, go to and follow the instructions there.

Re: [time-nuts] Raspberry Pi 4 oscillator replacement

[Trent Piepho]

On Thu, Feb 4, 2021 at 4:09 PM Hal Murray  wrote:
> In case anybody isn't familiar with ARM SOC chips, they typically have a layer
> of muxes between the external pins and the internal I/O devices.  I don't know
> if the chip used in the Pi-4 works this way.  Quite likely.
>
>
> The system designer has to find a combination of mux settings that works for
> the application.  Leftover pins can be used as GPIO.  And the clocking is
> tangled up in there.

While true for many pins, usually quite a few are not muxed.  Those
deemed necessary for all designs or with specialized drivers.  So
typically there will be no mux for power pins, dram interface, xtal
inputs, and high speed differential data, like USB3, PCIe, SGMII, MIPI
CSI or DSI.  Or perhaps muxing between different high speed serial
interfaces.

It seems to be very common for ARM SoC chips to be designed to be
clocked by either an XTAL and a built in oscillator, or via an
external clock generator connected to the same pin.  But, I've never
seen a SoC where this pin could be muxed.

I've not seen one where the input clock frequency had much of a range.
It might be 24-26 MHz, but never 10 - 52 MHz.  At least documented.
But the clock is invariably an input to a programmable PLL or PLLs,
and those PLLs are likely programmable with a rather wider range of
multipliers and dividers than required for the limited documented
input frequency range.

One might have better luck with a chip from Ti or NXP, since, unlike
Broadcom, they have documentation on how their clocks and PLLs work.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] An Other Ublox 8

[Trent Piepho]

On Thu, Jan 28, 2021 at 1:34 AM Javier Herrero  wrote:
> I have been playing lately with the M8F, but did not had a detailed look
> to the 30.72MHz frequency as John did. I have used it along with a
> Zynq-based board (a very unexpensive surplus EBAZ4205, that can be get
> for ~10 EUR) to made an ntp server, replacing the kernel clocksource
> from the ARM timer with an external timer implemented at the FPGA
> running from the 30.72MHz output (multiplied inside the FPGA to 250MHz),

Would this mean any timestamps generated via
clock_gettime(CLOCK_MONOTONIC_RAW) are going to be directly derived
from the value of your FPGA timer?  Because if that is the case, then
isn't the PPS phase offset from the system clock effectively the phase
offset of the GPS to the GPS?  I.e., it's zero because the same clock
is being compared to itself?

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] x86 CPU Timekeeping and clock generation

[Trent Piepho]

On Wed, Jan 6, 2021 at 6:26 AM Tom Holmes  wrote:
>
> Am I missing something or maybe I don't understand
> the situation , but I am under the impression that
> the RTC has it's own battery and crystal unrelated
> to the processor clock. Seems like in that case,
> the 24 MHz won't have any effect on the
> timekeeping drift.

It was like that, but the days of external RTC chips, e.g. from
Dallas, are largely over.  PC now have it integrated into the chipset.
Though it still has a power source and 32.768kHz xtal of its own.

In the embedded SoC world of phones, Raspberry PIs, and the OP's
Tinkerboard, there would virtually always be an RTC available in the
SoC with the CPU, or in the PMIC, or in both.  This would usually have
its own 32.768kHz xtal, but often there is an option to reduce the BoM
and use an internal RC oscillator instead of an external xtal or clock
signal, at greatly reduced accuracy.

The 32k xtal isn't for timekeeping accuracy, but for power savings.
In the lowest power modes the main PLLs will be shut down.  Certain
parts of the SoC will still be able to run using the 32kHz clock
domain and a low power output from the PMIC.

Linux doesn't use the RTC as the system clock.  During boot, the
kernel will usually set the system time from the RTC, and then the RTC
doesn't get used much, if at all.  There is a mode where the system
time is periodically copied back into the RTC, which can be enabled
with NTP.  And software like chrony has the ability to manage the RTC
and keep it in sync more intelligently, though I don't think any
distro uses this by default.

There are a number of sources that Linux can use as the system clock.
An on x86 system, it would usually be tsc, hpet, and acpi_pm.  On an
ARM board, arch_sys_counter.  The kernel subsystems for RTCs and for
clocksources are totally separate and the RTC, e.g. rtc-cmos, isn't a
clocksource.

Using adjtimex, it's possible to see what the current kernel
parameters are for clock adjustment.  The frequency adjustment would
need to be zero.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] x86 CPU Timekeeping and clock generation

[Trent Piepho]

On Tue, Jan 5, 2021 at 9:42 PM Luiz Paulo Damaceno
 wrote:
> The 24 MHz comes from an synthesizer that is locked to an atomic clock, the
> clock of NTP server (also 24 MHz, but an embedded board (Tinkerboard)) also
> comes from the same Atomic clock that is feeding other synthesizer for
> generates 24 MHz to this board.

The RK3288 has some PWM generators.  These are of course also fed from
PLLs derived from the same 24 MHz input.

So, why not produce a signal on the PWM that can be compared to your
reference?  This would tell you if the error is in the clock
generation on the SoC or something in software that happens afterward.
Or at least as far as the PWM clock tree overlaps the kernel
timesource clock tree, which could be the CPU clock but it can be
other things too.

> The experiment is the following: 1- synchronize the computer's clock to NTP
> server then leave it running free (no periodic synchronization), 2 -

NTP will set the frequency skew too, so even if it is not doing
periodic synchronization, there may still be a programmed frequency
skew.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Time Interval Counter(?) for high-precision watch measurement

[Trent Piepho]

I used the open source application tg, https://github.com/vacaboja/tg

I've found the microphone from some wired earbuds works well.  Just set
against the watch.  It picks up quartz watches.

tg lets you "calibrate" it by comparing to a reference audio PPS source,
like a quartz watch.  But it doesn't adjust for drift, e.g. from
temperature, while using it.  It would be better if it would use accurate
audio timestamps from a gps or ntp synced system time.  Which could be done.

On Tue, Sep 8, 2020 at 12:58 PM Warren Kumari  wrote:

> On Tue, Sep 8, 2020 at 3:15 PM djl  wrote:
> >
> > Sort of eavesdropping... you might try a piezoelectric pickup, available
> > cheap on epay for guitars, etc. Looking at the actual motion kick of the
> > hand(s)?
>
> I have one of these for mechanical watches: https://one-of.com/accuracy-2/
>
> It works nicely for tracking / accuracy of my mechanical watches - it
> uses a piezo for pickup, and hooks up to an iPhone / iPad (presents
> itself as a microphone). It reports on "accuracy" and amplitude, and
> allows me to save a "library" of watches so I can monitor them over
> time.
> Of course, this of for mechanical watches, and so "accuracy" is fairly
> relative ( COSC-certified chronometers must accurate to within -4/+6
> seconds a day).
>
> I just tried it with a Citizen Eco-Drive quartz (hoping that it might
> pick up the stepper "tick"), but it doesn't even detect that a watch
> is on the sensor
>
> W
>
> > Don
> >
> > On 2020-09-08 11:59, Raven L wrote:
> > > Hi Bob, thanks very much for all the info!
> > >
> > > You're right that my input signal is pretty crummy - if anything you
> > > overestimate it. Eyeballing it on the scope, the edge is not
> > > particularly
> > > defined at all - resolution better than a couple of microseconds just
> > > isn't
> > > possible. I've typically used gate times in the 10^4 to 10^5 seconds
> > > range
> > > (though typically collecting a measurement each second and doing the
> > > gating
> > > myself in software, as modern watches have multi-millisecond phenomenon
> > > that are interesting to study in the 1 to 100 second range).  I have
> > > tried
> > > picking up electrical signals from the actual contacts going from the
> > > IC to
> > > the motor, but even then the cleaner edge I got wasn't worth the
> > > trouble.
> > >
> > > I took an initial pass through the manuals of the models you mentioned,
> > > I
> > > really appreciate the guidance. (and the well-deserved correction on 2g
> > > tipover for AT crystals - I'm not sure what I was thinking when I wrote
> > > that but my notes confirm I was way off). I do hope to move up to a
> > > basic
> > > home lab running off CSAC and Rb standards in the next few years once
> > > I've
> > > seen all there is to see in wristwatches, but it looks like I have a
> > > lot to
> > > digest.
> > >
> > > One question if you know the answer on these 53131 and family models -
> > > the
> > > manual mentions that RS232 is talk-only to a printer. Are there
> > > reasonable
> > > tools out there for emulating a printer on a computer to pick up the
> > > data?
> > > Otherwise it looks like I would need to set up GPIB.
> > >
> > > On Tue, Sep 8, 2020, 1:12 PM Bob kb8tq  wrote:
> > >
> > >> Hi
> > >>
> > >> First off, 0.001 seconds per year is ~ 3x10^-11. If you are talking
> > >> about
> > >> the 2G tip effect on a typical AT cut crystal that’s up around 2 ppb.
> > >>
> > >> Next up, low frequency / small package crystals are (inevitably)
> > >> relatively
> > >> low Q devices. Low Q degrades ADEV performance / increases noise. If
> > >> you *could* measure a high Q device to 3x10^-11 in one second, it’s a
> > >> good bet that a low Q device will take 10X to 100X that amount of
> > >> time.
> > >>
> > >> The stepper motor in a watch is a low frequency inductive device. The
> > >> waveform out of it has a (very) limited bandwidth. Again another
> > >> factor
> > >> that
> > >> will stretch out the time involved in the measurement. Your pickup
> > >> coil
> > >> likely
> > >> also has some issues. ( That assumes the watch is still closed up. I
> > >> would
> > >> not
> > >> recommend opening one up for testing …).
> > >>
> > >> So far, none of this is looking at the frequency counter. We’re just
> > >> looking
> > >> at the device you are trying to measure.
> > >>
> > >> Since there does not appear to be a need to get the data really
> > >> quickly,
> > >> none of this is a show stopper. It simply suggests that something out
> > >> in
> > >> the hundreds of seconds is likely to be the sort of gate time
> > >> involved.
> > >>
> > >> If you are after 3x10^-11 on a 100 second gate, that comes out to a
> > >> rather
> > >> convenient 3 ns resolution. Better than that would be fine, but that’s
> > >> roughly
> > >> what you “need” to have.
> > >>
> > >> There are lots of low cost counters out there that will hit that sort
> > >> of
> > >> number.
> > >> The HP 5334 and HP 5335 both come to mind. They 

Re: [time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Mon, Jul 13, 2020 at 4:52 PM Hal Murray  wrote:
> > Is there any way for a USB device to synchronise with the CPU clock (perhaps
> > via the USB framing) so that a special-purpose device could timestamp the 
> > PPS
> > occurrence with respect to CPU time ?

It seems maybe this was thought about in USB 1.1.  From the UHCI spec:

Minor adjustments can be made to this value [ 1-ms frame time]
...  This feature permits minor changes to be made to the frame time
period, if necessary, to maintain real-time synchronization throughout
a USB system.

It seems like it would not be that hard to get the USB frame sequence
phase locked to the system clock.  One would need a way to measure the
phase offset of the USB S-o-F vs the system clock, and then it's a
standard process to phase lock, with the necessary control to do this
described above.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Mon, Jul 13, 2020 at 8:54 AM Petr Titěra  wrote:
>
> All Prolific  chips I saw claimed to be USB 2.0 Full-speed. That means
> they are polled only once in 1ms and there is no way how to change it
> (poll rate is selected at hardware level).

Looking at the UHCI specification, for USB 1.1 HCIs, while there poll
rate is fixed at about 1 ms, it is possible to tweak it slightly via
the start of frame modify register.

If one is measuring a PPS, then the pulses are not random.  It's
possible to try to align the USB frames with the PPS and do quite a
bit better than just a 1 msec poll rate would do at random.  Certainly
would be hard to fit this into the software stack.  And I don't see
this ability in the EHCI spec, though I'm certainly no expert with USB
HCI drivers, so perhaps it's gone now because no one could figure out
how to use it.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Sun, Jul 12, 2020 at 11:51 AM jimlux  wrote:

>
> Is the PPS via USB CTS stamped at interrupt time? or is it stamped
> higher up in the stack?
>
> I started tracing this out, but then decided I'm not going to be writing
> Linux USB drivers any time soon, so gave it up.
>
> I could easily imagine that the interrupt comes in, marks a thread as
> "ready to run" and the "oh CTS has changed state" is detected at a
> higher level.

Best way to see this is to do live tracing.  Use perf to turn on event
captures with IRQs, context switches, PPS events, USB events.  Then in
something like Trace Compass you can see the timeline of what is
actually happening.  The hard IRQ context first, then maybe a context
switch into a kernel thread that was made runnable by an action of the
irq handler.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Sun, Jul 12, 2020 at 5:08 AM Matthias Welwarsky
 wrote:

>
> If you think about using an AM3358, there's zero reason to use a GPIO for PPS
> input. There are much better options, like the gptimer inputs or the eCAP
> engine, which runs on a 200MHz clock and is therefore able to create much more
> accurate timestamps.

A much better way to go for sure. There would be just the latency of
the PPS vs the capture block's clock, plus anything in the logic of
that block.  All the delays that happen afterward, both just in
getting the interrupt to the CPU and also in handling the interrupt in
software before a PPS timestamp can be generated, can be subtracted
out by using the timer.

How much difference does it make?  I've actually got data on that, see
attached graph.  This is for a Cyclone V, but it's very comparable to
an AM3358, other than being dual core.  The uncorrected jitter
histogram is similar to what using a GPIO would give.  Though it uses
a custom PPS driver, which I wrote to try to do a good job at
timestamping the interrupt consistently.  The time measured here is
the difference of the PPS timestamp vs 1 second on the system clock.
This means we have:

Jitter in the Telit JN3 GPS PPS signal
Delay for PPS hardware in CycloneV to detect an edge.  I think it had
about a 200 MHz clock.
Jitter in the system clock for τ=1 sec
All the delays getting that interrupt to the CPU over AXI,
interrupting the ARM, querying the INTC, etc. that I've already gone
on about for some length.
Software time spent in kernel code (Interrupt handler, GIC driver, PPS
driver), before we get to create a timestamp.

Of course this is just the jitter in this latency.  The constant part
we don't actually know.  And this is usually about 1.5 µs, but can be
up to 100 μs.

The Corrected part of the histogram is what you'd get with a capture
mode timer.  In this case the clock was 50 MHz for that timer.  This
leaves the jitter from the first three sources: the GPS, the PPS
capture hardware, and the system clock.   And this puts us down in the
100 ns range, with a worse case of up to about 1 µs.

So that's what using a timer to capture a PPS can provide.  Two orders
of magnitude decrease in jitter.  We can also see that if one isn't
using a timer, the jitter in how long an interrupt handler takes to
generate a timestamp totally dominates the jitter from the GPS, system
clock, and a timer module.
___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Sat, Jul 11, 2020 at 7:37 PM jimlux  wrote:
>
> You might want to look into one of the "real time" linux kernels or
> other similar implementations - they might have "turned some of the
> knobs" to improve the handling of device data.

The real time kernel primarily is about trying to get closer to making
hard real time guarantees.  To put a hard maximum on latencies.  You
can't really do that on Linux, but the real time patches made it
better.  Mostly this is about finding a place where X could not happen
in time because the CPU was doing Y instead, and Y couldn't be
interrupted.  And find a way to interrupt Y so X doesn't have to wait.

One of the biggest changes was to thread all interrupts.  So that
interrupts can be interrupted.  If you are after low interrupt
latency, this actually makes things worse.  But it makes the worst
case much better than before, at the expense of the average case.


>
> There is a reason why USB support was late in coming to Linux compared
> to other devices. And there's a reason why everyone curses at serial

I thought Linux had USB support quite early.  I remember making a
cable so I could test the USB 1.0 in my 430HX based computer, the
first Intel chipset to have USB.  I believe it didn't work properly,
some kind of chipset problem.  Unless it was the 430FX, it's been so
long.  Google tells me the 430FX didn't have USB.  But I recall that
Intel's first chipset with USB had a broken controller and the
Ministry of Truth changed the advertising to "it never had USB."

A check shows there was Linux USB support in 1996, while Windows
didn't get it until 1997.  There was a big redesign of the Linux USB
stack by Linus in 1999.  I remember things working quite a bit better
after that was done.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Fri, Jul 10, 2020 at 12:17 PM Tim S  wrote:
>
> I believe the issue is arising from the understanding of "what" is being
> interrupted.
>
> When a UART or GPIO is triggering an interrupt - for the UART it's a FIFO
> notice that data is available to be pulled out, for the GPIO it's
> notification of a state change (and optionally, a specific state change:
> high-low/low-high).

On a processor like the Ti Sitara AM3358 found in the beaglebone or
the Broadcom processors in the various RPis, it's a little more
complex than that.  Both at the hardware level and at the software
level.

The UART IP block in the SoC has a module clock supplied from the
processor clock tree.  Likely two, one to drive the logic in the UART
and another to clock the external signals.  For the AM3358, there is a
100 MHz clock for the UART and an up to 48 MHz clock for the external
signals.

If the UART FIFO has reached a trigger point or the RTS line has
changed state, the UART will only raise its interrupt line on the edge
of the module clock, e.g. 100 MHz.  A GPIO controller would be the
same way.

That interrupt line does not go straight to the processor.  It goes to
an interrupt controller, such as the ARM GIC or a proprietary
controller.  The AM3358 uses a TI controller.  This controller deals
with priority of interrupts, masking, level/edge triggering (this is
about how the interrupt line from the UART or GPIO controller to the
INTC acts, not about how a GPIO will trigger an interrupt), steering
of interrupts to multiple CPU cores, and finally raising the interrupt
line in a CPU core.  Which might not happen immediately if another
interrupt is already executing.  The interrupt controller has its own
clock domain too, running at maybe 1/2 the processor's frequency.  It
takes a non-zero and non-constant amount of time for the INTC to pass
on the interrupt.

Once that happens, the CPU automatically does some stuff to respond to
the interrupt line.  It needs to finish executing the current CPU
instruction.  They don't all take one cycle, so this might be more or
less time.  The CPU also needs to jump to the main interrupt handler,
saving the current address.

We haven't even run one single software instruction yet, but we are
finally at the interrupt handler and have some control.  But it has
some stuff it must do first.  Save registers, switch stacks, etc.
That might access memory, which might not be cached, and so there's
more delay here.

It's got to figure out what interrupt occurred from the INTC.  This is
just a read from the memory mapped register.  But that means an AXI
bus transaction.  The CPU sends a request over the SoC interconnect
addressed to the interrupt controller.  The controller sends a
response back with the data.  Maybe there is contention on the AXI bus
due to data from other AXI masters, e.g. bus mastering DMA from the
USB controller.  More non-constant delays.

At this point the kernel interrupt handler will have figured out the
interrupt is from the UART and will call the handler registered by the
UART driver, GPIO driver or maybe a PPS driver.  What I've seen
discussed seems to treat this as the starting point of an interrupt.
But it's hardly the start.  We've gone through several IP blocks,
crossed a half dozen clock domains, sent and received messages via an
on chip network, and executed hundreds of instructions.

> When a USB PHY trips an interrupt it's to indicate that carrier data is
> ready, or for a USB MAC that burst blocks over the link are being decoded.
> There is an intermediate step between the USB carrier data being
> received/decoded, and being interpreted by the driver as a specific type of
> data - and the time that it is made available to the host OS.  Because that
> is handled in the driver, it is subject to all of the other host OS
> interrupts - so timing is not deterministic for the USB driver.

To figure out there was an interrupt from the UART, that it was due to
RTS changing, and do something about it, it takes quite a few
instructions.  And a number of register accesses that needs to do to
the UART, usually that means from over AXI to a AXI to APB interface
than then APB to a UART (I've never seen a UART as an AXI master, they
are always on a simpler peripheral bus like APB or OCP).  This takes
time.

Figuring out a USB interrupt was due to the completion of an interrupt
transfer, and that an interrupt is present in the end point, also
takes that same stuff.  Except the USB controller might be on AXI
directly.  It's quite a bit more complex.  But there isn't really a
fundamental difference on the software level.

___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.

Re: [time-nuts] Linear or quadractic fit algorithms for small microcontrollers?

[Trent Piepho]

On Thu, Dec 27, 2018 at 8:47 PM Tim Shoppa  wrote:

>
> Getting rid of the outliers turns out to be important when using the ES100
> in tracking mode. It turns out that in my noisy household environment here
> on the east coast, every few hundred attempts, the ES100 tracking attempt
> will return as a claimed success BUT it is off by way more than 100ms,
> often by an integer number of seconds (say 1-3 seconds).
>
>
I found a median filter worked well for this.   The Härdle and Steiger
algorithm is O(log n) time complexity and O(n) space complexity for a
filter of size N.  It doesn't need to be very big to remove outliers like
this.
___
time-nuts mailing list -- time-nuts@lists.febo.com
To unsubscribe, go to 
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.